arXiv: 2405.04434v5 [cs. CL] 19 Jun 2024

Qdeepseek

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model / DeepSeek-V2: 一个强大, 经济且高效的 MoE 语言模型

DeepSeek-AI

research@deepseek. com

Abstract

We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. It comprises 236B total parameters, of which 21B are activated for each token, and supports a context length of 128K tokens. DeepSeek-V2 adopts innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. MLA guarantees efficient inference through significantly compressing the Key-Value (KV) cache into a latent vector, while DeepSeekMoE enables training strong models at an economical cost through sparse computation. Compared with DeepSeek 67B, DeepSeek-V2 achieves significantly stronger performance, and meanwhile saves 42.5% of training costs, reduces the KV cache by 93.3%, and boosts the maximum generation throughput to 5.76 times. We pretrain DeepSeek-V2 on a high-quality and multi-source corpus consisting of 8.1T tokens, and further perform Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to fully unlock its potential. Evaluation results show that, even with only 21B activated parameters, DeepSeek-V2 and its chat versions still achieve top-tier performance among open-source models. The model checkpoints are available at [https://github. com/deepseek-ai/DeepSeek-V2](https://github. com/deepseek-ai/DeepSeek-V2).

推出 DeepSeek-V2: 强 MoE 语言模型, 训练省钱, 推理省缓存. 总参 236B, 每 token 激活 21B, 上下文 128K. 架构两件套: Multi-head Latent Attention(MLA)与 DeepSeekMoE. MLA 把 KV cache 压成潜变量, 推理更轻; DeepSeekMoE 靠稀疏计算把强模型训得起. 相对 DeepSeek 67B: 更强, 同时训练成本省 42.5%, KV cache 减 93.3%, 最大生成吞吐提到 5.76 倍. 预训练语料 8.1T, 再做 SFT 与 RL. 即使只激活 21B, 基座与 Chat 仍进开源第一梯队. 权重: https://github. com/deepseek-ai/DeepSeek-V2

解释: MLA= 把 Key/Value 联合压到低维潜向量再参与注意力, 推理主要缓存这份压缩向量, 而不是每头全量 K/V. DeepSeekMoE = 细粒度路由专家 + 隔离的共享专家, 用稀疏激活扩大总容量, 控制每步算力.

Figure 1 | (a) MMLU accuracy vs. activated parameters, among different open-source models. (b) Training costs and inference efficiency of DeepSeek 67B (Dense) and DeepSeek-V2.

图 1｜(a) 开源模型 MMLU 准确率对激活参数; (b) DeepSeek 67B(Dense)与 DeepSeek-V2 的训练成本与推理效率.

Contents

• 1 Introduction 4
• 2 Architecture 6
  • 2.1 Multi-Head Latent Attention: Boosting Inference Efficiency 6
    • 2.1.1 Preliminaries: Standard Multi-Head Attention 6
    • 2.1.2 Low-Rank Key-Value Joint Compression 7
    • 2.1.3 Decoupled Rotary Position Embedding 8
    • 2.1.4 Comparison of Key-Value Cache 8
  • 2.2 DeepSeekMoE: Training Strong Models at Economical Costs 9
    • 2.2.1 Basic Architecture 9
    • 2.2.2 Device-Limited Routing 9
    • 2.2.3 Auxiliary Loss for Load Balance 10
    • 2.2.4 Token-Dropping Strategy 11
• 3 Pre-Training 11
  • 3.1 Experimental Setups 11
    • 3.1.1 Data Construction 11
    • 3.1.2 Hyper-Parameters 12
    • 3.1.3 Infrastructures 12
    • 3.1.4 Long Context Extension 13
  • 3.2 Evaluations 13
    • 3.2.1 Evaluation Benchmarks 13
    • 3.2.2 Evaluation Results 14
    • 3.2.3 Training and Inference Efficiency 16
• 4 Alignment 16
  • 4.1 Supervised Fine-Tuning 16
  • 4.2 Reinforcement Learning 17
  • 4.3 Evaluation Results 18
  • 4.4 Discussion 20
• 5 Conclusion, Limitation, and Future Work 21
• A Contributions and Acknowledgments 27
• B DeepSeek-V2-Lite: A 16B Model Equipped with MLA and DeepSeekMoE 29

• B.1 Model Description 29
• B.2 Performance Evaluation 30
• C Full Formulas of MLA 31
• D Ablation of Attention Mechanisms 31
  • D.1 Ablation of MHA, GQA, and MQA 31
  • D.2 Comparison Between MLA and MHA 31
• E Discussion About Pre-Training Data Debiasing 32
• F Additional Evaluations on Math and Code 32
• G Evaluation Formats 33

1. Introduction

In the past few years, Large Language Models (LLMs) (Anthropic, 2023; Google, 2023; OpenAI, 2022, 2023) have undergone rapid development, offering a glimpse into the dawn of Artificial General Intelligence (AGI). In general, the intelligence of an LLM tends to improve as the number of parameters increases, allowing it to exhibit emergent capabilities across various tasks (Wei et al., 2022). However, the improvement comes at the cost of larger computing resources for training and a potential decrease in inference throughput. These constraints present significant challenges that impede the widespread adoption and utilization of LLMs. In order to tackle this problem, we introduce DeepSeek-V2, a strong open-source Mixture-of-Experts (MoE) language model, characterized by economical training and efficient inference through an innovative Transformer architecture. It is equipped with a total of 236B parameters, of which 21B are activated for each token, and supports a context length of 128K tokens.

过去几年大模型发展很快, 参数越大往往越强, 但训练算力与推理吞吐也一起变差. 为此推出开源 MoE 模型 DeepSeek-V2: 创新 Transformer 架构, 训练经济, 推理高效; 总参 236B, 每 token 激活 21B, 上下文 128K.

We optimize the attention modules and Feed-Forward Networks (FFNs) within the Transformer framework (Vaswani et al., 2017) with our proposed Multi-head Latent Attention (MLA) and DeepSeekMoE. (1) In the context of attention mechanisms, the Key-Value (KV) cache of the Multi-Head Attention (MHA) (Vaswani et al., 2017) poses a significant obstacle to the inference efficiency of LLMs. Various approaches have been explored to address this issue, including Grouped-Query Attention (GQA) (Ainslie et al., 2023) and Multi-Query Attention (MQA) (Shazeer, 2019). However, these methods often compromise performance in their attempt to reduce the KV cache. In order to achieve the best of both worlds, we introduce MLA, an attention mechanism equipped with low-rank key-value joint compression. Empirically, MLA achieves superior performance compared with MHA, and meanwhile significantly reduces the KV cache during inference, thus boosting the inference efficiency. (2) For Feed-Forward Networks (FFNs), we follow the DeepSeekMoE architecture (Dai et al., 2024), which adopts fine-grained expert segmentation and shared expert isolation for higher potential in expert specialization. The DeepSeekMoE architecture demonstrates great advantages compared with conventional MoE architectures like GShard (Lepikhin et al., 2021), enabling us to train strong models at an economical cost. As we employ expert parallelism during training, we also devise supplementary mechanisms to control communication overheads and ensure load balance. By combining these two techniques, DeepSeek-V2 features strong performance (Figure 1(a)), economical training costs, and efficient inference throughput (Figure 1(b)), simultaneously.

在 Transformer 里同时改注意力与 FFN: (1) MHA 的 KV cache 拖累推理; GQA/MQA 能省缓存却常掉分. MLA 做低秩 KV 联合压缩, 经验上强过 MHA, 同时大幅减推理 KV. (2) FFN 跟 DeepSeekMoE: 细粒度专家分割 + 共享专家隔离, 相对 GShard 一类常规 MoE 优势明显; 训练用专家并行时另加通信与负载均衡机制. 两件套叠在一起: 强(图 1(a)), 训练省, 推理快(图 1(b)).

We construct a high-quality and multi-source pre-training corpus consisting of 8.1T tokens. Compared with the corpus used in DeepSeek 67B (our previous release) (DeepSeek-AI, 2024), this corpus features an extended amount of data, especially Chinese data, and higher data quality. We first pretrain DeepSeek-V2 on the full pre-training corpus. Then, we collect 1.5M conversational sessions, which encompass various domains such as math, code, writing, reasoning, safety, and more, to perform Supervised Fine-Tuning (SFT) for DeepSeek-V2 Chat (SFT). Finally, we follow DeepSeekMath (Shao et al., 2024) to employ Group Relative Policy Optimization (GRPO) to further align the model with human preference and produce DeepSeek-V2 Chat (RL).

预训练语料 8.1T, 相对 67B 更大, 中文更多, 质量更高. 全量预训练后, 用 1.5M 会话(数学, 代码, 写作, 推理, 安全等)做 SFT, 得到 Chat (SFT); 再按 DeepSeekMath 用 GRPO 对齐人类偏好, 得到 Chat (RL).

解释: GRPO(组相对策略优化)= 同一题采样一组回答, 用组内相对奖励估优势, 省掉 PPO 里同规模的 critic.

We evaluate DeepSeek-V2 on a wide range of benchmarks in English and Chinese, and compare it with representative open-source models. Evaluation results show that even with only 21B activated parameters, DeepSeek-V2 still achieves top-tier performance among open-source models and becomes the strongest open-source MoE language model. Figure 1(a) highlights that, on MMLU, DeepSeek-V2 achieves top-ranking performance with only a small number of activated parameters. In addition, as shown in Figure 1(b), compared with DeepSeek 67B, DeepSeek-V2 saves 42.5% of training costs, reduces the KV cache by 93.3%, and boosts the maximum generation throughput to 5.76 times. We also evaluate DeepSeek-V2 Chat (SFT) and

中英多基准评测: 仅激活 21B 仍进开源第一梯队, 并成为当时最强开源 MoE. 图 1(a): MMLU 上激活不多也能排前列. 图 1(b): 相对 67B, 训练成本 −42.5%, KV −93.3%, 最大生成吞吐 ×5.76. Chat (SFT) 与

Figure 2 | Illustration of the architecture of DeepSeek-V2. MLA ensures efficient inference by significantly reducing the KV cache for generation, and DeepSeekMoE enables training strong models at an economical cost through the sparse architecture.

图 2｜DeepSeek-V2 架构示意. MLA 靠大幅减少生成期 KV cache 抬推理效率; DeepSeekMoE 靠稀疏结构把强模型训得起.

DeepSeek-V2 Chat (RL) on open-ended benchmarks. Notably, DeepSeek-V2 Chat (RL) achieves 38.9 length-controlled win rate on AlpacaEval 2.0 (Dubois et al., 2024), 8.97 overall score on MT-Bench (Zheng et al., 2023), and 7.91 overall score on AlignBench (Liu et al., 2023). The English open-ended conversation evaluations demonstrate that DeepSeek-V2 Chat (RL) has top-tier performance among open-source chat models. In addition, the evaluation on AlignBench indicates that in Chinese, DeepSeek-V2 Chat (RL) outperforms all of open-source models, and even beats most of closed-source models.

Chat (RL) 也做了开放生成评测: AlpacaEval 2.0 长度控制胜率 38.9, MT-Bench 总分 8.97, AlignBench 总分 7.91. 英文开放对话进开源 Chat 第一梯队; AlignBench 上中文超过全部开源, 并压过多数闭源.

In order to facilitate further research and development on MLA and DeepSeekMoE, we also release DeepSeek-V2-Lite, a smaller model equipped with MLA and DeepSeekMoE, for the open-source community. It has a total of 15.7B parameters, where 2.4B are activated for each token. Detailed descriptions about DeepSeek-V2-Lite can be found in Appendix B.

为方便社区继续研究 MLA 与 DeepSeekMoE, 另释出 DeepSeek-V2-Lite: 总参 15.7B, 每 token 激活 2.4B. 细节见附录 B.

In the rest of this paper, we first provide a detailed description of the model architecture of DeepSeek-V2 (Section 2). Subsequently, we introduce our pre-training endeavors, including the training data construction, hyper-parameter settings, infrastructures, long context extension, and the evaluation of model performance and efficiency (Section 3). Following this, we demonstrate our efforts in alignment, encompassing Supervised Fine-Tuning (SFT), Reinforcement

后文结构: §2 架构; §3 预训练(数据, 超参, 基建, 长上下文, 效果与效率); §4 对齐(SFT, 强化

Learning (RL), the evaluation results, and other discussion (Section 4). Finally, we summarize the conclusion, deliberate on the current limitations of DeepSeek-V2, and outline our future work (Section 5).

学习, 评测与讨论); §5 结论, 局限与未来工作.

2. Architecture 架构

By and large, DeepSeek-V2 is still in the Transformer architecture (Vaswani et al., 2017), where each Transformer block consists of an attention module and a Feed-Forward Network (FFN). However, for both the attention module and the FFN, we design and employ innovative architectures. For attention, we design MLA, which utilizes low-rank key-value joint compression to eliminate the bottleneck of inference-time key-value cache, thus supporting efficient inference. For FFNs, we adopt the DeepSeekMoE architecture (Dai et al., 2024), a high-performance MoE architecture that enables training strong models at an economical cost. An illustration of the architecture of DeepSeek-V2 is presented in Figure 2, and we will introduce the details of MLA and DeepSeekMoE in this section. For other tiny details (e. g., layer normalization and the activation function in FFNs), unless specifically stated, DeepSeek-V2 follows the settings of DeepSeek 67B (DeepSeek-AI, 2024).

整体仍是 Transformer: 每块注意力 + FFN. 注意力用 MLA(低秩 KV 联合压缩, 去掉推理 KV 瓶颈); FFN 用 DeepSeekMoE(经济地训强模型). 总览见图 2. 其余细节(层归一化, FFN 激活等)未特别说明处跟 DeepSeek 67B.

2.1. Multi-Head Latent Attention: Boosting Inference Efficiency Multi-Head Latent Attention: 抬推理效率

Conventional Transformer models usually adopts Multi-Head Attention (MHA) (Vaswani et al., 2017), but during generation, its heavy Key-Value (KV) cache will become the bottleneck that limit the inference efficiency. In order to reduce the KV cache, Multi-Query Attention (MQA) (Shazeer, 2019) and Grouped-Query Attention (GQA) (Ainslie et al., 2023) are proposed. They require a smaller magnitude of KV cache, but their performance does not match MHA (we provide the ablation of MHA, GQA and MQA in Appendix D. 1).

常规 Transformer 用 MHA, 生成时沉重 KV cache 会卡吞吐. MQA, GQA 能少存 KV, 但性能跟不上 MHA(消融见附录 D. 1).

For DeepSeek-V2, we design an innovative attention mechanism called Multi-head Latent Attention (MLA). Equipped with low-rank key-value joint compression, MLA achieves better performance than MHA, but requires a significantly smaller amount of KV cache. We introduce its architecture in the following, and also provide a comparison between MLA and MHA in Appendix D. 2.

V2 设计 MLA: 低秩 KV 联合压缩, 强过 MHA, KV 却少得多. 下文讲结构; 与 MHA 对照见附录 D. 2.

2.1.1. Preliminaries: Standard Multi-Head Attention 预备: 标准 Multi-Head Attention

We first introduce the standard MHA mechanism as background. Let 𝑑 be the embedding dimension, $n _ { h }$ be the number of attention heads, $d _ { h }$ be the dimension per head, and $\mathbf { h } _ { t } \in \mathbb { R } ^ { \tilde { d } }$