Petuun-从统计机器学习视角理解深度学习：算法、理论与可扩展计算（英文）-2018.9-286页-13mb

Summary of "A Statistical Machine Learning Perspective of Deep Learning: Algorithm, Theory, Scalable Computing"

Core Content

This document provides a comprehensive overview of deep learning from a statistical machine learning perspective, focusing on the interplay between deep learning (DL) and probabilistic graphical models (PGMs). It explores the theoretical foundations, algorithmic components, and computational mechanisms that underpin both fields, emphasizing their similarities and differences.

Main Points

1. Statistical and Algorithmic Foundations of Deep Learning

• Deep learning is viewed through the lens of statistical learning, particularly in terms of representation, learning, and inference.
• The joint probability distribution over multiple variables is a central concept in probabilistic modeling.
• The number of state configurations can be extremely large, but PGMs offer a way to reduce this complexity by encoding dependencies.

2. Overview of Deep Learning Components

• Activation functions: Include linear, ReLU, sigmoid, and tanh, which are used to introduce non-linearity into neural networks.
• Layers: Such as fully connected, convolutional, recurrent, and ResNets, which form the building blocks of deep networks.
• Loss functions: Such as cross-entropy and mean squared error, which guide the learning process.
• Reverse-mode automatic differentiation (backpropagation): The core algorithm for computing gradients in deep networks, enabling efficient optimization.

3. Graphical Models vs. Neural Networks

• Graphical models (GMs) are used to encode probabilistic relationships and uncertainty in a structured way.
• Neural networks are primarily function approximators, though some, like Boltzmann machines and sigmoid belief networks, can be considered as proper GMs.
• GMs are more interpretable and allow for structured inference algorithms, while deep networks are often black-box and rely on gradient-based methods.

4. Probabilistic Graphical Models: Basics

• GMs represent joint probability distributions using graphs, where nodes represent variables and edges represent dependencies.
• Directed GMs (Bayesian Networks) encode causal relationships, while undirected GMs (Markov Random Fields) represent correlations.
• The joint distribution can be factored into a product of simpler terms, significantly reducing the number of parameters.

5. Restricted Boltzmann Machines (RBMs)

• RBMs are undirected graphical models with a bipartite graph structure.
• They are used to model the joint distribution of visible and hidden variables.
• Training involves computing the gradient of the log-likelihood using two phases: the positive phase (clamped) and the negative phase (unclamped), often approximated via MCMC sampling.
• RBMs can be seen as infinitely deep belief networks due to their layered structure and tied weights.

6. Sigmoid Belief Networks (SBNs)

• SBNs are Bayesian networks with binary variables and conditional probabilities represented by sigmoid functions.
• They exhibit the "explain away effect," where conditioning on one variable can cause dependencies among others.
• Learning and inference in SBNs are typically done via Monte Carlo methods like Gibbs sampling.

7. Deep Belief Networks (DBNs)

• DBNs are hybrid models combining multiple RBMs in a chain structure.
• They are trained in a greedy, layer-wise fashion, with each layer pre-trained independently before fine-tuning.
• While this approach is effective for representation learning, it lacks proper joint training and can lead to suboptimal models.

8. Bayesian Learning of Neural Networks

• Bayesian learning provides a framework for incorporating uncertainty into neural network parameters.
• Techniques such as deep kernel learning are explored to integrate domain knowledge and improve model robustness.

9. Comparative Analysis

• DL vs. ML (e.g., GMs):
  • Empirical goal: DL focuses on tasks like classification and feature learning, while ML (e.g., GMs) emphasizes latent variable inference and transfer learning.
  • Structure: Both use graphical structures, but DL emphasizes computational graphs.
  • Algorithm: DL primarily uses backpropagation, while ML involves a variety of inference and learning algorithms.
  • Evaluation: DL evaluates on end performance, while ML can evaluate on intermediate quantities.
  • Implementation: DL requires many tricks for optimization, while ML has more standardized approaches.
  • Experiments: DL often uses massive real data, while ML experiments are typically on simulated data.

Key Information

• Representation: PGMs provide a structured way to represent complex distributions, reducing the number of parameters needed.
• Learning: DL relies on gradient descent and backpropagation, while PGMs use techniques like EM, message passing, and MCMC.
• Inference: DL uses forward passes, while PGMs involve complex algorithms for approximating distributions.
• Challenges: DBNs face issues with exact inference due to the "explain away" effect, and their training is often ad-hoc.
• Scalability: DL is well-suited for large-scale data and complex models, but the lack of structured inference algorithms is a limitation.

Conclusion

The document highlights the strengths and limitations of both deep learning and probabilistic graphical models, advocating for a unified framework that combines the best of both worlds. It emphasizes the importance of understanding the theoretical underpinnings of DL and how it can be integrated with PGMs for more robust and interpretable models.