【DeepSeek】V3技术报告_53页_1mb

DeepSeek-V3 Technical Report Summary

Core Content

DeepSeek-V3 is a large-scale Mixture-of-Experts (MoE) language model with 671B total parameters, of which 37B are activated per token. It is designed to achieve efficient inference and cost-effective training, making it a competitive open-source model in both performance and resource usage.

Main Contributions

• Innovative Load Balancing Strategy: DeepSeek-V3 introduces an auxiliary-loss-free load balancing approach to maintain expert load balance without compromising model performance. This strategy dynamically adjusts bias terms to balance the load during training.
• Multi-Token Prediction (MTP) Training Objective: This training objective enhances the model's ability to predict multiple future tokens at once, improving data efficiency and enabling better representation for future tokens. It also supports speculative decoding for inference acceleration.
• Efficient Training Framework: DeepSeek-V3 utilizes FP8 mixed precision training, which reduces GPU memory usage and accelerates training. It also implements DualPipe for efficient pipeline parallelism and computation-communication overlap to minimize communication overhead.

Key Architectural Components

2.1 Basic Architecture

• Multi-Head Latent Attention (MLA): MLA compresses attention keys and values to reduce the KV cache size while maintaining performance comparable to standard Multi-Head Attention (MHA).
• DeepSeekMoE: This MoE architecture uses finer-grained experts and shared experts to enhance training efficiency. It is optimized for expert parallelism across 8 nodes.
• No Token-Dropping: The model maintains a good load balance, so no tokens are dropped during training or inference.

2.2 Multi-Token Prediction

• Implementation Details: MTP is implemented using D sequential modules, each with a shared embedding layer, output head, Transformer block, and projection matrix.
• Training Objective: The MTP loss is calculated by averaging the cross-entropy loss across all prediction depths and multiplying it by a weighting factor λ.

Infrastructure and Optimization

3.1 Compute Clusters

• Trained on a cluster with 2048 H800 GPUs.
• Each node contains 8 GPUs connected via NVLink and NVSwitch.
• InfiniBand (IB) is used for cross-node communication to ensure high bandwidth utilization.

3.2 Training Framework

• Utilizes 16-way Pipeline Parallelism (PP), 64-way Expert Parallelism (EP), and ZeRO-1 Data Parallelism (DP).
• Implements DualPipe to reduce pipeline bubbles and hide communication during training.
• Uses efficient cross-node all-to-all communication kernels to fully utilize InfiniBand and NVLink bandwidths.
• Achieves minimal memory overhead without relying on tensor parallelism.

3.3 FP8 Training

• Mixed Precision Framework: FP8 is used for both computation and storage to reduce memory usage and accelerate training.
• Improved Precision: The model benefits from quantization and multiplication in FP8, maintaining performance while reducing resource costs.
• Low-Precision Communication: FP8 is also used for communication to further enhance efficiency.

3.4 Inference and Deployment

• Prefilling and Decoding: Efficient strategies are used for prefilling and decoding to optimize inference performance.
• Hardware Design Suggestions: The paper provides insights into communication and compute hardware design to support future large-scale models.

Pre-Training

• Data Construction: Trained on 14.8 trillion diverse and high-quality tokens.
• Training Stability: The training process is remarkably stable, with no irrecoverable loss spikes or rollbacks.
• Cost Efficiency: Training DeepSeek-V3 on each trillion tokens requires only 180K H800 GPU hours (about 3.7 days on a 2048 GPU cluster).
• Total Training Cost: Only 2.788M H800 GPU hours are needed for full training, costing $5.576M at $2 per GPU hour.

Post-Training

• Supervised Fine-Tuning (SFT): Aligns the model with human preferences.
• Reinforcement Learning (RL): Further enhances performance through reward model and group relative policy optimization.
• Knowledge Distillation: Reasoning capabilities are distilled from the DeepSeek-R1 series into DeepSeek-V3, improving its reasoning performance.
• Self-Rewarding: The model is fine-tuned using its own output as a reward signal.

Evaluation Results

• Knowledge Benchmarks:
  • Outperforms other open-source models on MMLU, MMLU-Pro, and GPQA, achieving 88.5, 75.9, and 59.1, respectively.
  • Performance is comparable to leading closed-source models like GPT-4o and Claude-3.5-Sonnet.
• Factuality Benchmarks:
  • Demonstrates superior performance in Chinese factual knowledge on SimpleQA and Chinese SimpleQA.
• Code and Math:
  • Achieves state-of-the-art performance on math-related benchmarks, even outperforming o1-preview on MATH-500.
  • Leads in coding competition benchmarks like LiveCodeBench.
• Reasoning and Generation:
  • Shows strong reasoning capabilities and competitive generation performance.
  • Maintains control over output style and length during post-training.

Limitations and Future Directions

• The model is not perfect and has some limitations in performance compared to certain closed-source models.
• Future work includes improving reasoning accuracy, enhancing long-context generation, and further optimizing training efficiency.

Conclusion

DeepSeek-V3 represents a significant advancement in the field of open-source large language models. By combining efficient architecture, auxiliary-loss-free load balancing, multi-token prediction, and FP8 training, it achieves strong performance with low training costs. It is currently the strongest open-source base model in code and math tasks, and its inference and deployment strategies ensure scalable and efficient use. The model's checkpoints are publicly available at https://github.com/deepseek-ai/DeepSeek-V3.