杜克大学在读博士生温伟-云端与边缘AI系统的深度学习_45页_11mb

Deep Learning in Cloud-Edge AI Systems

Core Content Overview

This document discusses the integration of deep learning techniques in cloud-edge AI systems, focusing on challenges and solutions for efficient training and inference. It introduces TernGrad, a method for reducing communication overhead in distributed deep learning, and Structurally Sparse Deep Neural Networks (SSDNNs), which aim to reduce model size and improve inference speed by learning structured sparsity.

---

Main Points and Key Information

1. Cloud vs. Edge Deep Learning

• Cloud Deep Learning:

  • Utilizes parallelism and large computing resources.
  • Faces communication bottlenecks due to the exchange of large gradients.
  • High computational cost but efficient communication.

• Edge Deep Learning:

  • Emphasizes model size and inference speed.
  • Requires efficient and compact models for deployment on resource-constrained devices.
  • Direct sparse convolution and structurally sparse DNNs are proposed for better performance on edge devices.

---

2. TernGrad: Ternary Gradients for Communication Reduction

• Concept:

  • TernGrad quantizes gradients to three levels: {-1, 0, 1} (i.e., <2 bits).
  • Reduces communication cost in distributed training.

• Mathematical Formulation:

  • $\tilde{\pmb{g}}_t = s_t \cdot \mathit{sign}(\pmb{g}_t) \circ \pmb{b}_t$
  • $s_t \triangleq ||\pmb{g}t||\infty$
  • $P(b_{tk} = 1 | \boldsymbol{g}t) = |g{tk}| / s_t$

• Convergence:

  • TernGrad almost-truly converges under similar assumptions as standard SGD.
  • Stronger gradient bound compared to standard SGD, leading to better convergence.

• Performance:

  • TernGrad is simple to implement and can be integrated with various optimizers (e.g., SGD, Momentum, Adam).
  • Hyperparameter tuning can reduce the accuracy gap.
  • Performance improvements are more significant with larger worker counts and smaller communication bandwidth.

---

3. Structurally Sparse Deep Neural Networks (SSDNNs)

• Definition:

  • SSDNNs remove weights or connections in groups (e.g., rows, columns, blocks, or entire layers).
  • This leads to reduced storage and computational requirements.

• Regularization Method:

  • Group Lasso is used to learn structured sparsity.
  • Weights are grouped, and the regularization is applied to each group.

• Implementation:

  • Easy to implement using TensorFlow.
  • Example code snippet for adding group Lasso regularization.

• Performance Results:

  • SSDNNs achieve similar accuracy to dense networks while significantly improving speed.
  • For example, in the case of AlexNet:
    • Top-1 error remains around 44.66% with structured sparsity.
    • Speedup on CPU and GPU is significant, especially for column and row sparsity.

---

4. Efficiency of SSDNNs

• Speedup:

  • SSDNNs offer good speedup when using customized hardware.
  • Speedup is more pronounced for models with more fully-connected layers.

• Inefficiency on CPUs:

  • Sparse matrix formats (e.g., CSR) are not as efficient on CPUs as they are on GPUs.
  • This highlights the need for optimized hardware for sparse operations.

• Non-Structured Sparsity:

  • Non-structured sparsity can lead to inefficiency in both CPU and GPU environments.

---

5. Research Highlights

• Distributed Training:

  • TernGrad: Ternary gradients reduce communication in distributed deep learning.
  • Ongoing work on large-batch training.

• Efficient Inference:

  • Structurally sparse DNNs, lower-rank DNNs, and compact DNNs are proposed.
  • These models maintain high accuracy while being more efficient.

---

6. Performance Comparison

• Accuracy and Speed:

  • SSDNNs maintain high accuracy (e.g., GoogLeNet accuracy loss < 2%).
  • Speedup is observed across different models and datasets.

• Examples:

  • LeNet: Close accuracy and small variance with TernGrad.
  • AlexNet: Speedup with structured sparsity and efficient inference.

---

7. Conclusion

• TernGrad effectively reduces communication overhead in distributed deep learning without significant accuracy loss.
• Structurally sparse DNNs offer efficient inference and reduced model size, making them suitable for edge deployment.
• Both methods are promising for improving the efficiency of AI systems in cloud-edge environments.