【DeepSeek】2024年-V2模型技术报告经济高效的混合专家语言模型

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

Core Content

DeepSeek-V2 is a state-of-the-art Mixture-of-Experts (MoE) language model designed for efficient training and inference. It features a total of 236B parameters, with only 21B activated per token, and supports a context length of up to 128K tokens. This model is built on an innovative Transformer architecture that includes Multi-head Latent Attention (MLA) and DeepSeekMoE, which together enhance performance while reducing computational and memory costs.

Main Points

• Model Overview:

  • Total parameters: 236B
  • Activated parameters per token: 21B
  • Context length: 128K tokens
  • Model checkpoints available at https://github.com/deepseek-ai/DeepSeek-V2

• Inference Efficiency:

  • MLA reduces the Key-Value (KV) cache by 93.3% compared to DeepSeek 67B.
  • This is achieved through low-rank key-value joint compression, which compresses keys and values into a latent vector, significantly decreasing memory usage during inference.
  • Additionally, decoupled rotary position embedding (RoPE) is used to maintain position sensitivity without coupling with the compressed vectors, allowing efficient KV cache management.

• Training Efficiency:

  • DeepSeek-V2 saves 42.5% of training costs compared to DeepSeek 67B.
  • It achieves a 5.76 times increase in maximum generation throughput.
  • The DeepSeekMoE architecture enables sparse computation, allowing the model to train efficiently while maintaining strong performance.

• Architecture Innovations:

  • MLA (Multi-head Latent Attention): Reduces KV cache while maintaining or improving performance over MHA.
  • DeepSeekMoE: Implements fine-grained expert segmentation and shared expert isolation, improving expert specialization and reducing redundancy.
  • Additional mechanisms like device-limited routing, auxiliary losses for load balance, and token-dropping strategy ensure efficient training and resource utilization.

• Training Data:

  • A high-quality, multi-source corpus of 8.1T tokens is used for pre-training.
  • This corpus includes a larger amount of Chinese data, improving performance on Chinese tasks.
  • The data is filtered to remove contentious content and improve quality, ensuring a more balanced and representative training set.

• Fine-Tuning and Alignment:

  • The model undergoes Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to align with human preferences.
  • DeepSeek-V2 Chat (SFT) and DeepSeek-V2 Chat (RL) are evaluated on various benchmarks.
  • On AlpacaEval 2.0, DeepSeek-V2 Chat (RL) achieves a 38.9% length-controlled win rate.
  • On MT-Bench and AlignBench, it scores 8.97 and 7.91, respectively, showing strong performance in both English and Chinese.

• Model Variants:

  • DeepSeek-V2-Lite: A smaller version with 15.7B total parameters and 2.4B activated per token, designed for easier deployment and research.

• Performance:

  • Despite only activating 21B parameters per token, DeepSeek-V2 outperforms other open-source models in performance.
  • It is the strongest open-source MoE language model according to evaluation results.
  • In Chinese tasks, DeepSeek-V2 Chat (RL) surpasses most closed-source models.

Key Information

• MLA reduces the KV cache by 93.3% while maintaining strong performance.
• DeepSeekMoE enables sparse computation, leading to 42.5% lower training costs.
• Inference throughput is 5.76 times higher than DeepSeek 67B.
• The model is trained on 8.1T tokens, with a focus on Chinese data and data quality.
• Fine-tuning is done using 1.5M conversational sessions across multiple domains.
• DeepSeek-V2 Chat (RL) performs exceptionally well on both English and Chinese benchmarks, including AlignBench and MT-Bench.
• DeepSeek-V2-Lite is released for research and deployment purposes.

Summary

DeepSeek-V2 represents a significant advancement in the field of large language models, particularly in the realm of Mixture-of-Experts (MoE) architectures. By introducing MLA and DeepSeekMoE, it achieves a balance between strong performance and efficient resource utilization. The model is trained on a high-quality, multi-source corpus of 8.1T tokens, with an emphasis on Chinese content and data filtering to reduce bias. It is evaluated on various benchmarks and shows top-tier performance even with only 21B activated parameters per token. Additionally, a lighter variant, DeepSeek-V2-Lite, is made available for broader accessibility and research. The combination of low-rank compression, sparse routing, and load balancing strategies ensures that DeepSeek-V2 is both cost-effective and highly performant.