DeepSeek模型关键创新技术综述_11页_939kb

Summary of DeepSeek Models' Key Innovative Techniques

Core Content

DeepSeek-V3 and DeepSeek-R1 are leading open-source Large Language Models (LLMs) that achieve performance comparable to state-of-the-art closed-source models like GPT and Claude, while using significantly fewer training resources. This paper reviews the key innovative techniques behind their success and highlights open research questions.

Main Techniques

1. Refined Transformer Architecture

• Multi-Head Latent Attention (MLA):

  • MLA reduces the KV cache size by decomposing the projection matrices into lower-rank matrices.
  • The down-projection matrix $W^{DKV}$ compresses both keys and values into a latent vector $\mathbf{c}_t^{KV}$.
  • The up-projection matrices $W^{UK}, W^{UV}$ are used to reconstruct keys and values during inference.
  • An additional decoupled RoPE is introduced to improve performance, though no ablation study has been conducted on this component.

• Decoupled Rotary Position Embedding (RoPE):

  • Separates RoPE into queries and keys, enhancing the model's ability to handle long contexts.
  • Reduces the KV cache size compared to standard MHA, with specific formulas for cache size and memory usage.

• Low-Rank Key-Value Joint Compression:

  • Further compresses queries using a low-rank matrix $W^{DQ}$ and up-projection $W^{UQ}$.
  • This strategy reduces activation memory during training, improving efficiency.

2. Mixture of Experts (MoE)

• Fine-Grained Expert Segmentation:

  • Segments each FFN into smaller experts, increasing the total number of experts and activated experts.
  • Enhances combinatorial flexibility and model performance.

• Shared Expert Isolation:

  • Introduces shared experts to capture common knowledge, reducing parameter redundancy.
  • Maintains a constant computational cost by adjusting the number of routed experts.

• Load Balancing:

  • An auxiliary loss is used to balance expert utilization across different tokens.
  • A bias term is introduced to the affinity score to achieve better load balance without affecting model performance.
  • A sequence-wise auxiliary loss is also used to avoid extreme imbalance within a single sequence.

3. Multi-Token Prediction (MTP)

• MTP predicts multiple tokens in a causal chain, improving sample efficiency during training.
• The training objective is the average of cross-entropy losses across different depths.
• However, MTP introduces additional training time overhead, which is not fully addressed in the ablation study.

4. Co-design of Algorithms, Frameworks, and Hardware

• DualPipe:

  • A pipeline parallelism algorithm that overlaps computation and communication.
  • Divides computation into forward and backward chunks, reducing pipeline bubbles.
  • Can be optimized with a "cut-in-half" procedure to eliminate the bidirectional part.

• FP8 Mixed Precision Training:

  • Uses FP8 precision for most operations to speed up training while maintaining accuracy.
  • Certain operators (e.g., embedding, normalization) are kept in FP32 to ensure numerical stability.
  • A fine-grained quantization strategy extends the dynamic range of FP8.

5. Group Relative Policy Optimization (GRPO)

• GRPO is an efficient variant of Proximal Policy Optimization (PPO), eliminating the need for a value model.
• It directly estimates the advantage using group rewards from the old policy.
• Two types of advantage estimation are used: outcome supervision and process supervision.
• GRPO maximizes a training objective that includes both advantage terms and KL divergence regularization for policy alignment.

6. Post-Training Techniques

• Pure Reinforcement Learning (RL):

  • DeepSeek-R1-Zero is trained with pure RL without SFT data.
  • It achieves continuous performance improvement and naturally develops reasoning behaviors.
  • However, it faces issues such as poor readability and language mixing.

• Iterative Training with SFT and RL:

  • DeepSeek-R1 uses an iterative approach alternating between SFT and RL.
  • Includes stages like Cold Start, Reasoning-oriented RL, Rejection Sampling, and RL Alignment.
  • Each stage is designed to enhance different aspects of the model's performance and alignment with human preferences.

Key Information

• DeepSeek models demonstrate high performance with reduced training costs.
• MLA and MTP significantly improve efficiency and performance.
• MoE with shared experts and load balancing strategies enhance scalability and utilization.
• GRPO is a promising RL method for LLMs, especially in scenarios with sparse rewards.
• The co-design of algorithms, frameworks, and hardware enables efficient large-scale training.
• Post-training techniques like iterative SFT-RL and GRPO are critical for aligning models with human preferences and improving reasoning capabilities.

Open Questions and Research Opportunities

• The theoretical justification of the load balancing loss function remains unclear.
• The impact of the decoupled RoPE on model performance needs further investigation.
• The training overhead of MTP and its potential improvements are open research areas.
• The effectiveness of the GRPO algorithm in different LLM architectures and tasks requires more study.
• The integration of multiple post-training techniques for optimal performance and alignment is an ongoing challenge.