DeepSeek-V3技术报告（英）-2025_53页_1mb

DeepSeek-V3 Overview and Key Innovations

DeepSeek-V3 is a 671B parameter Mixture-of-Experts (MoE) language model developed as an open-source alternative to closed-source models, achieving high performance at reduced costs. Trained on 14.8T tokens, it incurs only $5.576M in H800 GPU rental costs, making it economically viable for scaling. The model integrates specialized architectures and optimizations to balance resource efficiency with performance.

Architecture Highlights
The core design combines Multi-Head Latent Attention (MLA) for efficient inference and DeepSeekMoE for cost-effective training. MLA uses latent compression to reduce memory footprints during attention mechanisms. The MoE component employs an auxiliary-loss-free load balancing strategy, avoiding performance degradation while maintaining expert activation efficiency. Additionally, a Multi-Token Prediction (MTP) objective is introduced to predict multiple future tokens per input, enhancing performance on benchmarks by densifying training signals and supporting speculative decoding.

Training Infrastructure and Optimization
Training is optimized through FP8 mixed-precision frameworks, enabling accelerated computation and memory savings. The DualPipe algorithm is used for pipeline parallelism, minimizing communication overhead by overlapping computation and data movement. Custom kernels leverage InfiniBand (IB) and NVLink for cross-node all-to-all communication, achieving substantial speedups on up to 2048 GPUs. FP8 training is validated for low-precision stability, with fine-grained quantization and high-precision accumulation strategies mitigating numerical errors.

Pre-Training Process
The model is trained on a diverse dataset with enhanced ratios for math and code-related samples, expanding multilingual coverage beyond English and Chinese. Long-context capabilities are achieved through YaRN scaling, extending context from 4K to 128K tokens in two stages. The process is stable, with no irrecoverable loss spikes, requiring only 2.788M GPU hours total (2.664M during pre-training).

Post-Training and Model Distillation
Post-training aligns the model with human preferences via supervised fine-tuning and reinforcement learning, incorporating knowledge distillation from the DeepSeek-R1 series to enhance reasoning capabilities. Reinforcement learning uses a rule-based and model-based reward system, along with Group Relative Policy Optimization (GRPO), to refine generation quality. Distillation from DeepSeek-R1 transfers strong reasoning skills into standard architectures without compromising model size.

Evaluation Results
Comprehensive benchmark tests show DeepSeek-V3 surpassing other open-source and closed-source models in key areas like educational knowledge (e.g., MMLU-Pro), coding competitions (e.g., LiveCodeBench), mathematics (e.g., MATH), and language understanding. It achieves performance on par with leading closed-source models like GPT-4o and Claude-3.5-Sonnet, while demonstrating cost advantages. The model was also evaluated in open-ended tasks like AlpacaEval and Arena-Hard, achieving milestone performances that bridge the gap between open-source and commercial models.

Conclusion and Future Directions
DeepSeek-V3 represents a strong open-source base model with significant cost savings and performance achievements. Limitations include deployment overheads, but planned future work includes further optimization for infinite context lengths, advanced architectures, data quality enhancements, and expert reasoning capability extensions, alongside hardware-focused recommendations for reduced communication bottlenecks and higher FP8 accumulation precision.