Qdeepseek

DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression / DeepSeek-V4.1-Flash: 把 KV Cache 压缩推到极限

DeepSeek-AI research@deepseek. com

DeepSeek-AI . research@deepseek. com

Abstract

The widespread adoption of long-horizon agents has made model workloads increasingly inputheavy. Although prior work has substantially reduced the cost of long-context computation, prefill remains computationally expensive, and large KV caches continue to strain HBM and SSD capacity and data-transfer bandwidth. Together, these compute, storage, and bandwidth demands constitute the primary bottleneck to further lowering deployment costs. To address this challenge, we introduce DeepSeek-V4.1-Flash, a multimodal Mixture-of-Experts (MoE) model with 552B backbone parameters and support for contexts of up to one million tokens. With its Causal Encoder-Decoder (CED) architecture, the model activates 16B parameters per token during decode but only 8B parameters during prefill, substantially improving cost efficiency for agentic workloads. To push the limits of KV cache compression, DeepSeek-V4.1- Flash combines cross-layer KV cache reuse in Compressed Sparse Attention 2 (CSA2) with FP4 KV caching. These designs reduce its global KV cache footprint (always in HBM) to 890 bytes per token, roughly 1/4 of the corresponding footprint of DeepSeek-V4-Flash. Further, through a dedicated deployment optimization known as SWA Bounded Replay, DeepSeek-V4.1-Flash reduces its persistent KV cache footprint (always on SSD or in host memory) to roughly 1/8 of that of DeepSeek-V4-Flash. Despite its much smaller KV cache footprint, the model delivers substantially better performance than the baseline. In addition, we streamline the DeepSeek-V4 architecture and introduce several efficient architectural extensions. We pretrain DeepSeek-V4.1- Flash on a multimodal corpus comprising 45T tokens and conduct comprehensive post-training, yielding strong performance across diverse text-based and multimodal agentic scenarios. Model checkpoints are available at [https://huggingface. co/deepseek-ai/DeepSeek-V4.1-Flash](https://huggingface. co/deepseek-ai/DeepSeek-V4.1-Flash).

长程 Agent 普及后, 负载越来越偏输入侧. 长上下文算力已压过一轮, 但 prefill 仍贵, 巨大的 KV cache 继续挤占 HBM, SSD 容量与传输带宽-- 算力, 存储, 带宽合在一起, 成了继续降部署成本的主瓶颈. 为此推出 DeepSeek-V4.1-Flash: 多模态 MoE, 骨干 552B 参, 上下文最长 100 万 token. Causal Encoder-Decoder(CED) 让 decode 每 token 激活 16B, prefill 仅 8B, 更贴合输入偏重的 Agent 场景. KV 压缩靠 Compressed Sparse Attention 2(CSA2) 的跨层 KV 复用, 再加 FP4 KV cache: 常驻 HBM 的 global KV 压到每 token 890 字节, 约为 DeepSeek-V4-Flash 的 1/4; 部署侧再靠 SWA Bounded Replay, 常驻 SSD/主机内存的 persistent KV 约压到 V4-Flash 的 1/8. Cache 更小, 整体表现却明显强于基线. 同时精简 V4 架构并加入若干高效扩展; 在约 45T token 多模态语料上预训练, 再做完整后训练, 文本与多模态 Agent 场景均有强表现. 权重: https://huggingface. co/deepseek-ai/DeepSeek-V4.1-Flash.

Figure 1 | (a) Performance of DeepSeek-V4.1-Flash and its counterparts on agentic benchmarks. (b) Global KV cache size per token (in bytes) across generations of DeepSeek models, highlighting DeepSeek’s sustained efforts to reduce context memory requirements. DeepSeek-V4.1-Flash achieves approximately 4-fold and 437-fold reductions in per-token global KV cache size relative to DeepSeek-V4-Flash and DeepSeek-V1, respectively.

图 1｜(a) DeepSeek-V4.1-Flash 与对照模型在 Agent 基准上的表现. (b) DeepSeek 各代每 token 的 global KV cache 字节数, 可见持续压上下文内存; 相对 V4-Flash 与 V1, V4.1-Flash 的每 token global KV 约分别降到 1/4 与约 1/437.

Contents

• 1 Introduction 4

• 2 Architecture 7

  • 2.1 Overview 7
    • 2.1.1 Multimodal Architecture 8
  • 2.2 Causal Encoder-Decoder (CED) 9
  • 2.3 Compressed Sparse Attention 2 (CSA2) 9
    • 2.3.1 Cross-Layer KV and Index Reuse 10
    • 2.3.2 Hierarchical Sparse Indexer 11
  • 2.4 Efficient Architectural Extensions 12
    • 2.4.1 Single-Pass mHC 12
    • 2.4.2 Engram 13
    • 2.4.3 DSpark 13
    • 2.4.4 FP4 Main KV Cache 14
  • 2.5 Optimization 14

• 3 General Infrastructures 16

  • 3.1 Training Infrastructure 16
    • 3.1.1 Multimodal Training Infrastructure 16
    • 3.1.2 Attention Sharing Training for CSA2 17
    • 3.1.3 Engram 18
  • 3.2 Inference System 18
    • 3.2.1 Persistent KV Cache Management 19
    • 3.2.2 SWA Bounded Replay 20

• 4 Pre-Training 20

  • 4.1 Data Construction 20
  • 4.2 Pre-Training Setups 21
    • 4.2.1 Model Setups 21
    • 4.2.2 Training Setups 22
  • 4.3 Evaluations 23
    • 4.3.1 Evaluation Benchmarks 23
    • 4.3.2 Evaluation Results 23

• 5 Post-Training 25

• 1 引言 4

• 2 架构 7

  • 2.1 总览 7
    • 2.1.1 多模态架构 8
  • 2.2 Causal Encoder-Decoder(CED) 9
  • 2.3 Compressed Sparse Attention 2(CSA2) 9
    • 2.3.1 跨层 KV 与索引复用 10
    • 2.3.2 分层稀疏 Indexer 11
  • 2.4 高效架构扩展 12
    • 2.4.1 Single-Pass mHC 12
    • 2.4.2 Engram 13
    • 2.4.3 DSpark 13
    • 2.4.4 FP4 主 KV Cache 14
  • 2.5 优化 14

• 3 通用基础设施 16

  • 3.1 训练基础设施 16
    • 3.1.1 多模态训练基础设施 16
    • 3.1.2 CSA2 注意力共享训练 17
    • 3.1.3 Engram 18
  • 3.2 推理系统 18
    • 3.2.1 Persistent KV Cache 管理 19
    • 3.2.2 SWA Bounded Replay 20

• 4 预训练 20

  • 4.1 数据构建 20
  • 4.2 预训练设定 21
    • 4.2.1 模型设定 21
    • 4.2.2 训练设定 22
  • 4.3 评测 23
    • 4.3.1 评测基准 23
    • 4.3.2 评测结果 23

• 5 后训练 25

• 5.1 Post-Training Pipeline 25

  • 5.1.1 Large-Scale Agent Task Synthesis 25
  • 5.1.2 RL in Synthesized Tasks 26
  • 5.1.3 Running Agents at Massive Scale: DSec 27
  • 5.1.4 Controllable Reasoning Effort in RL 29

• 5.2 Asynchronous Post-training Infrastructure 30

  • 5.2.1 Overall Workflow 30
  • 5.2.2 Mitigating Length Bias and Off-Policy Effects 31
  • 5.2.3 Performance Optimization 31
  • 5.2.4 Large-Scale On-Policy Distillation 31

• 5.3 Evaluation 32

  • 5.3.1 Evaluation Setup 32
  • 5.3.2 Evaluation Results 33
  • 5.3.3 Performance across reasoning efforts 34
  • 5.3.4 Performance across agent scaffolds 34
  • 5.3.5 Multi-Agent 35

• 6 Conclusion, Limitations, and Future Directions 37

• A Author List 46

• B Evaluation Details 47

  • B.1 Scaffold Configurations 47
  • B.2 Reasoning Efforts across Scaffolds 48
  • B.3 Detailed Results of Reasoning Benchmarks across Reasoning Efforts 48

• C Exponential Token Penalty in Reasoning Effort Control 49

  • 5.1 后训练流水 25
    • 5.1.1 大规模 Agent 任务合成 25
    • 5.1.2 合成任务上的 RL 26
    • 5.1.3 大规模跑 Agent: DSec 27
    • 5.1.4 RL 中可控推理力度 29
  • 5.2 异步后训练基础设施 30
    • 5.2.1 总体流程 30
    • 5.2.2 缓解长度偏置与 off-policy 31
    • 5.2.3 性能优化 31
    • 5.2.4 大规模 On-Policy Distillation 31
  • 5.3 评测 32
    • 5.3.1 评测设定 32
    • 5.3.2 评测结果 33
    • 5.3.3 不同推理力度下的表现 34
    • 5.3.4 不同 Agent scaffold 上的表现 34
    • 5.3.5 多智能体 35

• 6 结论, 局限与未来方向 37

• A 作者名单 46

• B 评测细节 47

  • B.1 Scaffold 配置 47
  • B.2 各 Scaffold 上的推理力度 48
  • B.3 各推理力度下的推理基准细表 48

• C 推理力度控制中的指数 token 惩罚 49

1. Introduction

Applications of long-horizon agents have expanded rapidly in recent years, making ultra-longcontext processing an increasingly important model workload. Supporting such workloads requires not only efficient long-sequence processing, but also the persistent storage, reuse, and transfer of large KV caches. KV cache management has therefore become a foundational capability for model deployment, while introducing substantial challenges across computation, storage, and communication. Prior advances in sparse attention (DeepSeek-AI, 2025, 2026b) have significantly reduced the computational cost of long-sequence processing, making persistent storage and data movement increasingly prominent bottlenecks.

长程 Agent 应用近年扩张很快, 超长上下文变成越来越重要的负载. 支撑这类负载, 既要高效处理长序列, 也要持久存, 复用, 搬运大块 KV cache. KV 管理因此成了部署底座能力, 同时在算力, 存储, 通信上带来硬约束. 稀疏注意力(DeepSeek-AI, 2025, 2026b)已明显压低长序列计算成本, 持久存储与数据搬运反而更显眼.

Specifically, DeepSeek-V4 (DeepSeek-AI, 2026b) combines a global attention branch spanning the full context with local Sliding-Window Attention (SWA). The global branch maintains global KV, comprising main KV and indexer K, while SWA maintains local KV states. For a fixed window size, SWA KV storage is bounded independently of sequence length. For sufficiently long sequences, global KV therefore dominates the runtime KV footprint, which is constrained by HBM capacity. In addition, certain KV are persisted for prefix reuse, referred to as persistent KV caches, which are constrained by SSD and host memory capacity. I/O and interconnect bandwidth also limit cache migration and loading. Together, these constraints limit serving throughput, increase deployment costs, and ultimately hinder the deployment and adoption of agents over longer task horizons and across broader application scenarios.

具体说, DeepSeek-V4 把覆盖全上下文的全局注意力分支, 与局部 Sliding-Window Attention(SWA) 绑在一起. 全局分支维护 global KV(含 main KV 与 indexer K), SWA 维护局部 KV. 窗口固定时, SWA KV 存储与序列长度解耦; 序列足够长时, runtime KV(受 HBM 容量约束)主要由 global KV 主导. 另有一部分 KV 为前缀复用而持久化, 称 persistent KV, 受 SSD 与主机内存约束. I/O 与互联带宽也限制 cache 迁移与加载. 这些合在一起压吞吐, 抬部署成本, 最终拖累更长任务视界, 更广场景的 Agent 落地.

Further reducing the KV cache footprint is therefore critical to alleviating storage and communication bottlenecks and lowering the cost of long-context serving. To this end, we develop DeepSeek-V4.1-Flash, a multimodal Mixture-of-Experts (MoE) model designed for more aggressive KV cache compression. DeepSeek-V4.1-Flash has 552B backbone parameters, natively supports multimodal inputs, and accommodates contexts of up to one million tokens. We adopt a Causal Encoder-Decoder (CED) architecture, in which decoder global KV is projected from the final encoder hidden states. This design enables the model to activate 8B parameters per token during prefill and 16B during decode, which is particularly cost-effective for input-heavy agentic scenarios. Despite being considerably larger than DeepSeek-V4-Flash, DeepSeek-V4.1- Flash requires only approximately 1/4 as much runtime KV cache storage and 1/8 as much persistent KV cache storage at the same sequence length. Moreover, DeepSeek-V4.1-Flash delivers better overall performance than DeepSeek-V4-Flash.

继续压 KV 占用, 是缓解存储与通信瓶颈, 压低长上下文服务成本的关键. 于是做出 DeepSeek-V4.1-Flash: 面向更激进 KV 压缩的多模态 MoE. 骨干 552B, 原生多模态, 上下文最长 100 万 token. 采用 CED: decoder 的 global KV 由 encoder 末层隐状态投影得到, 从而 prefill 每 token 激活 8B, decode 激活 16B, 对输入偏重的 Agent 尤其省. 尽管规模明显大于 V4-Flash, 同序列长度下 runtime KV 约只要 1/4, persistent KV 约只要 1/8, 且整体表现更好.

This level of KV cache compression is achieved through joint optimizations in model architecture, cache precision, and deployment strategy. Conceptually, DeepSeek-V4 can be viewed as an SWA-based local-processing backbone augmented with compressed global context. This perspective motivates us to focus on simplifying the global branch while largely preserving the local attention design. At the architectural level, we design Compressed Sparse Attention 2 (CSA2), which applies cross-layer reuse to global KV (including main KV and indexer K) and Top-K indices to substantially reduce KV cache storage. CSA2 has three statically assigned modes: Full, Reindex, and Reuse. Full Mode generates global KV and performs indexing. Reindex Mode reuses the global KV from a preceding layer, and uses its own indexer Q to rescore the shared indexer K and select fresh Top-K indices. Reuse Mode reuses both global KV and the Top-K indices in a preceding layer, and directly performs sparse attention. In all three modes, each layer retains its own global Q and SWA KV. Sharing global KV and indexer K reduces duplicated cache storage. In addition, different from DeepSeek-V4 that employs the Compressed Sparse Attention (CSA)–Heavily Compressed Attention (HCA) hybrid architecture, DeepSeek-V4.1-Flash uses pure CSA2. At the cache-precision level, we use FP4 global KV caches during training with only marginal performance degradation. Together, CSA2 and FP4 KV

这档压缩来自架构, cache 精度与部署策略的联合优化. 概念上, V4 可看成「SWA 本地骨干 + 压缩全局上下文」; 于是重点简化全局分支, 局部注意力大体保留. 架构上设计 CSA2: 对 global KV(含 main KV 与 indexer K)以及 Top-K 索引做跨层复用, 大幅省存储. CSA2 静态分三种模式:Full 生成 global KV 并做索引; Reindex 复用前层 global KV, 用本层 indexer Q 重打共享 indexer K 并选新 Top-K; Reuse 复用前层 global KV 与 Top-K, 直接稀疏注意力. 三模式都保留本层 global Q 与 SWA KV. 共享 global KV 与 indexer K 减少重复缓存. 与 V4 的 CSA–HCA 混合不同, V4.1-Flash 走纯 CSA2. 精度上训练期用 FP4 global KV, 性能损失很小. CSA2 与 FP4 KV

Figure 2 | Single-token Decode FLOPs versus context length across generations of DeepSeek models. We account for compute precision by weighting BF16, FP8, and FP4 operations by 1, 0.5, and 0.25, respectively. DeepSeek-V4.1-Flash maintains nearly constant Decode FLOPs as context length increases, substantially reducing the computational cost of long-context scenarios.

图 2｜各代 DeepSeek 单 token Decode FLOPs 随上下文长度变化. 按精度加权: BF16, FP8, FP4 分别按 1, 0.5, 0.25. V4.1-Flash 在上下文变长时 Decode FLOPs 几乎持平, 长上下文算力成本明显下降.

caching reduce global KV cache storage to approximately 1/4 of that of DeepSeek-V4-Flash, as shown in Figure 1(b). At the deployment level, DeepSeek-V4.1-Flash, like DeepSeek-V4, uses Sliding-Window Attention (SWA) in every layer. In DeepSeek-V4, we use a hybrid strategy to balance the storage cost of persisting SWA KV caches against the computation required for exact reconstruction. Exact reconstruction requires replaying the most recent $L \times n _ { \mathrm { w i n } }$ tokens, where 𝐿 is the number of layers and $n _ { \mathrm { w i n } }$ is the SWA window size. In DeepSeek-V4.1-Flash, we introduce SWA Bounded Replay, which approximately reconstructs the required SWA KV states by replaying only the most recent $n _ { \mathrm { w i n } }$ tokens. Our experiments show that this incurs only negligible performance degradation. This finding establishes a new storage–computation trade-off, allowing us to avoid persisting SWA KV cache to SSD while incurring a small amount of prefill recomputation. With SWA Bounded Replay, the persistent KV cache footprint is further reduced to approximately 1/8 of that of DeepSeek-V4-Flash. Together, these optimizations greatly ease pressure on HBM and SSD capacity, reduce deployment costs, and pave the way for deployment at a larger scale.

二者合起来把 global KV 存到约 V4-Flash 的 1/4(图 1(b)). 部署上与 V4 一样每层都有 SWA. V4 用混合策略权衡「持久化 SWA KV 的存储」与「精确重建的计算」-- 精确重建要回放最近 $L \times n_{\mathrm{win}}$ 个 token($L$ 层数, $n_{\mathrm{win}}$ 窗口). V4.1-Flash 提出 SWA Bounded Replay: 只回放最近 $n_{\mathrm{win}}$ 个 token, 近似重建所需 SWA KV; 实验显示性能损失可忽略. 这给出新的存储–计算折中: 不必把 SWA KV 持久化到 SSD, 只付少量 prefill 重算. 配合后, persistent KV 进一步约到 V4-Flash 的 1/8. 合起来明显缓解 HBM/SSD 压力, 压低部署成本, 为更大规模服务铺路.

Complementing CED and CSA2, we further streamline the original DeepSeek-V4 architecture. Additionally, we upgrade the original mHC (Xie et al., 2026) design to Single-Pass mHC, with an accompanying Mega-mHC deployment kernel that halves activation memory traffic relative to the original four-kernel implementation. Furthermore, we integrate the Engram (Cheng et al., 2026b) conditional memory module to strengthen model capabilities. We also introduce the DSpark (Cheng et al., 2026a) speculative decoding architecture to improve decoding efficiency through semi-autoregressive draft generation and confidence-scheduled verification. With all the architectural improvements combined, the single-token Decode FLOPs of DeepSeek-V4.1- Flash remain nearly constant across context lengths. Figure 2 shows that extending the context length 256-fold, from 4K to 1M, increases its Decode FLOPs by only 1/4, significantly less than the growth observed for DeepSeek-V4-Flash.

在 CED 与 CSA2 之外, 继续精简 V4 架构: 把 mHC 升为 Single-Pass mHC, 配套 Mega-mHC 部署核, 相对原四核实现把激活内存流量减半; 接入 Engram 条件记忆增强能力; 引入 DSpark 投机解码(半自回归草稿 + 按置信度调度校验)抬解码效率. 架构改完后, 单 token Decode FLOPs 跨上下文几乎恒定. 图 2: 上下文从 4K 拉到 1M(×256), Decode FLOPs 只增约 1/4, 远小于 V4-Flash 的增长.

To fully realize the KV cache compression benefits of these architectural designs and further improve training and inference efficiency, we systematically co-optimize the training infrastructure and inference system for DeepSeek-V4.1-Flash, ensuring efficient and scalable large-scale multimodal training and long-context deployment. Training infrastructure supports disaggregated vision-encoder execution, balanced image sharding for long sequences, and cross-stage shared-state management for attention reuse. The inference system implements Encoder and De-

为把架构上的 KV 压缩吃满, 并继续抬训练/推理效率, 训练基础设施与推理系统系统性共优化: 训练侧支持视觉编码器解耦执行, 长序列均衡图像分片, 跨流水级共享状态以服务注意力复用; 推理侧实现 Encoder 与 De-

coder SWA Bounded Replay paths. Further optimizations include communication–computation overlap, sharded Engram embedding tables, and inference kernel fusion. In particular, each CSA2 Reuse Mode layer executes with only 15 kernels during prefill and 11 during decode. We also separate long-lived global KV storage from short-lived encoder SWA KV in host memory, using bounded replay to approximately reconstruct missing encoder SWA states.

coder 的 SWA Bounded Replay 路径; 再叠加通信–计算重叠, 分片 Engram 表, 推理核融合. 特别地, CSA2 Reuse 层 prefill 仅 15 个核, decode 仅 11 个. 主机内存上把长寿 global KV 与短寿 encoder SWA KV 拆开, 缺 SWA 时用有界回放近似重建.

During pre-training, we train DeepSeek-V4.1-Flash on a large-scale multimodal corpus comprising 45T tokens. Sparse attention is trained from scratch at a sequence length of 64K, without any dense attention warmup stages. After pre-training, the model possesses native multimodal capabilities and supports contexts of up to one million tokens. In our evaluations, DeepSeek-V4.1-Flash-Base achieves world knowledge, reasoning and coding abilities comparable to DeepSeek-V4-Pro-Base, and delivers 5%–10% improvements on held-out evaluations, using only 1/3 total parameters and 1/4 activated parameters. Together, these results highlight its strong parameter efficiency and reflect improvements in training data quality for real-world deployment.

预训练在约 45T token 多模态语料上完成; 稀疏注意力从 64K 序列长度直接训起, 无稠密注意力预热. 训完后原生多模态, 上下文可达 100 万 token. 评测上, V4.1-Flash-Base 的世界知识, 推理与代码能力可比 V4-Pro-Base, held-out 上再抬 5%–10%, 却只用约 1/3 总参, 1/4 激活参-- 参数效率与面向真实部署的数据质量提升一并可见.

Building on this base model, we conduct post-training to elicit its reasoning and agentic capabilities. In contrast to the architectural innovations described above, our post-training introduces no algorithmic innovation: the recipe follows the standard paradigm of supervised fine-tuning (SFT) followed by reinforcement learning (RL) and on-policy distillation (OPD), without any modification beyond well-established practice used in DeepSeek-V4 development (DeepSeek-AI, 2026b). All substantive changes lie instead in the data pipeline. We develop large-scale automated pipelines for data synthesis and environment construction, and progressively scale the data, tasks, and rollouts employed during RL, thereby extending the model’s capabilities across textual, multimodal, and agentic domains. Figure 1(a) summarizes DeepSeek-V4.1-Flash’s performance on core agentic benchmarks. Our evaluation shows that, despite its compact size, DeepSeek-V4.1-Flash exhibits a distinctive capability profile:

在 Base 上做后训练, 拉出推理与 Agent 能力. 与架构创新不同, 后训练没有新算法: 仍是 SFT → RL → on-policy distillation(OPD), 不超出 V4 已成熟做法(DeepSeek-AI, 2026b). 实质变化全在数据流水: 大规模自动合成与环境构建, 并逐步放大 RL 用的数据, 任务与 rollout, 覆盖文本, 多模态与 Agent. 图 1(a) 汇总核心 Agent 基准. 评测显示, 体量虽紧, 能力画像仍鲜明:

• Reasoning. The model delivers strong reasoning ability, sustaining high accuracy on reasoning-intensive benchmarks such as mathematics and competitive programming, showing comparable performance with top open-source models, such as Kimi-K3(Team et al., 2026a) and DeepSeek-V4-Pro.

• 推理. 数学与竞赛编程等推理密集基准上准确率高, 可比顶级开源如 Kimi-K3, DeepSeek-V4-Pro.

Agent. DeepSeek-V4.1-Flash achieves performance on par with closed-source frontier models across standard agentic benchmarks like Terminal-Bench 2.1 (Merrill et al., 2026), DeepSWE v1.1 (DataCurve, 2026), and AutomationBench (Shepard and Salimans, 2026). It has proven fully capable of handling everyday coding tasks and white-collar workflows. However, a gap with giant models remains on science-oriented agentic tasks, such as Terminal-Bench 4.0 (Marten et al., 2026a), that require expert-level domain knowledge.

Agent. 在 Terminal-Bench 2.1, DeepSWE v1.1, AutomationBench 等标准 Agent 基准上与闭源前沿持平; 日常编码与白领流程已能扛. 但面向科学, 需专家域知识的 Agent 任务(如 Terminal-Bench 4.0)与巨型模型仍有差距.

• Multimodal. Within the multimodal domain, the model surpasses top-tier open-source competitors like Kimi-K3 specifically on benchmarks evaluating visual reasoning and the interpretation of professional charts. Beyond formal metrics, it also exhibits practical utility in real-world visual agentic workflows, such as frontend development and office automation, where it can utilize rendered screen captures for visual inspection and self-correction. Nevertheless, we acknowledge that a distinct overall performance gap remains when compared to giant closed-source systems.

• 多模态. 视觉推理与专业图表解读类基准上超过 Kimi-K3 等顶级开源; 前端开发, 办公自动化等真实视觉 Agent 流程里, 能用渲染截图做检视与自纠. 相对巨型闭源, 整体仍有可感差距.

These results indicate that DeepSeek-V4.1-Flash can already match closed-source frontier models on the vast majority of benchmarks, and is capable of completing over 95% of real-world tasks. Meanwhile, its small activation footprint yields low inference latency and serving cost. We therefore believe that DeepSeek-V4.1-Flash offers a favorable trade-off between capability and efficiency, and can serve as a fast, affordable assistant supporting the daily work of a

多数基准上已能对齐闭源前沿, 真实任务完成率作者估计超 95%; 激活小, 延迟与服务成本低. 能力与效率折中有利, 适合当快速, 便宜的日常助手, 服务广大

Figure 3 | Overall architecture of DeepSeek-V4.1-Flash. The 40-layer network is divided into a causal encoder and a decoder, each with 20 layers. All feed-forward layers use standard DeepSeekMoE. The first two encoder layers use sliding window attention (SWA); the rest use Compressed Sparse Attention 2 (CSA2), with CSA2(ratio, mode) specifying the compression ratio and mode. The model also uses Single-Pass mHC, Engram, DSpark, and a Hierarchical Sparse Indexer.

图 3｜DeepSeek-V4.1-Flash 总体架构. 40 层网络分成因果 encoder 与 decoder 各 20 层. FFN 全用标准 DeepSeekMoE. Encoder 前两层纯 SWA, 其余用 CSA2(标注压缩比与模式). 另含 Single-Pass mHC, Engram, DSpark 与分层稀疏 Indexer.

broad population of users. In summary, DeepSeek-V4.1-Flash simultaneously improves model intelligence and inference efficiency while reducing deployment costs. It substantially lowers the cost barrier to deploying long-horizon agents at scale and creates new opportunities for their adoption across a broader range of scenarios. DeepSeek-V4.1-Flash also serves as a new starting point for our continued scaling efforts. Building on this foundation, we will pursue the joint scaling of model architecture, pre-training, and post-training to further explore the frontier of model intelligence.

用户. 总结: 智能, 推理效率与部署成本一并改善, 长程 Agent 规模化门槛明显降低, 应用面更宽. 它也是后续缩放的新起点-- 架构, 预训练, 后训练联合放大, 继续探智能边界.

2. Architecture 架构

2.1. Overview

DeepSeek-V4.1-Flash is a multimodal mixture-of-experts (MoE) Transformer that takes images and text as input and generates text autoregressively. Its language backbone comprises 40 causal Transformer layers, organized into a 20-layer causal encoder followed by a 20-layer decoder. Each layer incorporates both global attention and sliding window attention (SWA), except for the first two layers, which use SWA only. A vision encoder and an MLP projector convert images into visual embeddings that are processed jointly with text embeddings, with multimodal data incorporated from the start of language-model pre-training. Overall, DeepSeek-V4.1-Flash has 552B backbone parameters and 196B Engram parameters, activating 8B parameters per token during prefill and 16B during decode. Figure 3 illustrates the overall architecture of DeepSeek-V4.1-Flash.

DeepSeek-V4.1-Flash 是多模态 MoE Transformer: 图文进, 文本自回归出. 语言骨干 40 层因果 Transformer, 前 20 层因果 encoder, 后 20 层 decoder. 除前两层仅 SWA 外, 每层同时有全局注意力与 SWA. 视觉编码器 + MLP 投影把图变成视觉嵌入, 与文本嵌入一起处理; 多模态数据从语言模型预训练一开始就接入. 合计骨干 552B, Engram 196B; prefill 每 token 激活 8B, decode 激活 16B. 图 3 为总览.

The Causal Encoder–Decoder (CED) architecture and Compressed Sparse Attention 2 (CSA2)

CED 与 CSA2

address complementary costs of long-context inference. CED constructs the decoder’s global key-value (KV) cache from encoder outputs, allowing most prompt tokens to bypass full decoder computation while retaining layer-local sliding-window attention. This nearly halves prefill computation, lowering the cost of processing new or uncached inputs in agentic workloads with growing contexts. CSA2 shares global KV across layers to reduce cache storage and reuses sparse selections to reduce indexing work. In the decoder, a Hierarchical Sparse Indexer restricts later indexers to a candidate pool selected by an earlier indexer, further reducing the number of entries scored per query.

分别对付长上下文推理的互补成本. CED 用 encoder 输出构造 decoder 的 global KV, 多数 prompt token 可跳过完整 decoder 计算, 同时保留层内 SWA--prefill 算力近乎减半, Agent 语境下处理新/未缓存输入更便宜. CSA2 跨层共享 global KV 省存储, 复用稀疏选择省索引; decoder 里 Hierarchical Sparse Indexer 让后续 indexer 只在更早 indexer 选出的候选池里搜, 进一步压每 query 打分条目数.

We retain the shared and fine-grained routed experts of DeepSeekMoE (Dai et al., 2024), and introduce modality-specific load balancing (Wang et al., 2024a) for image and text tokens. Single-Pass mHC (Xie et al., 2026) revises residual-stream mixing to enable more efficient kernel fusion, and Engram (Cheng et al., 2026b) adds sparsely accessed conditional memory. We omit the MTP module during backbone pre-training and use DSpark (Cheng et al., 2026a) for speculative decoding. We train DSpark separately after the backbone pre-training stage. Additionally, we compress the main KV cache to FP4 to further reduce storage overhead. The following sections describe these components and the corresponding optimization changes.

保留 DeepSeekMoE 的共享专家与细粒度路由专家, 并对图/文 token 引入模态专用负载均衡. Single-Pass mHC 改残差流混合以便更高效核融合; Engram 加稀疏访问的条件记忆. 骨干预训练省略 MTP, 投机解码改用 DSpark(骨干训完后单独训). 主 KV 再压到 FP4. 下文分述各组件与优化改动.

2.1.1. Multimodal Architecture 多模态架构

The multimodal input pathway comprises a vision encoder and an MLP projector. For each input image, the vision encoder produces a spatial grid of visual features. A 3 × 3 pixelunshuffle operation then rearranges each local neighborhood along the channel dimension, reducing the spatial resolution before the MLP projector maps the features to the hidden dimension of the language backbone. Finally, the resulting visual embeddings are inserted at the corresponding image-token positions in the input embedding sequence and processed jointly with text embeddings by the language backbone.

多模态输入路径: 视觉编码器 + MLP 投影. 每张图先出空间特征网格; 再做 3×3 pixel-unshuffle(把局部邻域重排到通道维, 降空间分辨率), MLP 再映射到语言骨干隐维; 视觉嵌入插到对应 image token 位置, 与文本嵌入一并进骨干.

DeepSeek-ViT We train a vision encoder named DeepSeek-ViT from scratch to natively process images at varying resolutions. We build DeepSeek-ViT on the Vision Transformer (Dosovitskiy et al., 2021) architecture with several modifications. To accommodate inputs of arbitrary resolutions, we replace standard absolute positional embeddings with 2D-RoPE. To align the ViT more closely with LLM design principles, we replace the patch embedding layer’s convolution with a linear projection to ensure compatibility with the Muon optimizer. We also adopt RMSNorm (Zhang and Sennrich, 2019) for normalization and SwiGLU (Shazeer, 2020) as the activation function. Before feeding visual features into the LLM, we apply a pixel-unshuffle operation with 3 × 3 downsampling to reduce the visual token count by a factor of nine, effectively supporting input resolutions up to approximately 1344 × 1344 pixels.

DeepSeek-ViT 从零训的视觉编码器, 原生吃可变分辨率. 基于 ViT, 改动包括: 绝对位置嵌入换成 2D-RoPE; patch 嵌入的卷积改线性投影, 以兼容 Muon; 归一化用 RMSNorm, 激活用 SwiGLU. 进 LLM 前 3×3 pixel-unshuffle, 视觉 token 数 ÷9, 有效支持约 1344×1344.

Multimodal Auxiliary-loss-free Load Balancing for MoEs Image and text tokens exhibit distinct representation distributions and may induce different expert-routing preferences in MoEs. Balancing their aggregate load may therefore obscure modality-specific imbalance. To address this issue, we extend auxiliary-loss-free load balancing (Wang et al., 2024a) by maintaining separate expert-wise correction biases for text and image tokens. During routing, each token uses the correction biases associated with its modality for expert selection, while the original routing scores are retained for weighting the selected expert outputs. After each training step, the two sets of biases are updated independently according to their respective expert loads. This design balances expert utilization within each modality and contributes to stable and efficient multimodal training.

多模态无辅助损失负载均衡 图文表示分布不同, 可能诱导不同专家偏好; 只按总量均衡会掩盖模态内失衡. 于是扩展无辅助损失负载均衡: 为文, 图各维护一套专家校正偏置. 选专家时用本模态偏置, 加权仍用原始路由分数; 每步后两套偏置按各自负载独立更新. 模态内利用更均衡, 多模态训练更稳, 更高效.

2.2. Causal Encoder-Decoder (CED)

2.2. Causal Encoder-Decoder(CED)

In agentic workflows, frequent tool calls generate extensive prefill requests, imposing severe computational overhead when KV caches miss. To alleviate this prefill bottleneck, we propose the Causal Encoder-Decoder (CED) architecture, inspired by YoCo (Sun et al., 2024). YoCo reduces prefill computation by allowing the upper half of the layers to directly share the KV cache generated by the lower half. Building upon this concept, CED introduces a series of structural improvements to enhance both the overall KV cache capacity and the computational depth of KV generation. Consequently, CED successfully reduces nearly half of the prefill computation while maintaining performance comparable to the baseline.

Agent 流程里工具调用频繁, KV miss 时大量 prefill 很贵. CED 受 YoCo 启发: YoCo 让上半层直接共享下半层生成的 KV, 压 prefill. CED 在此之上做结构改进, 同时抬整体 KV 容量与 KV 生成的计算深度, 于是 prefill 近乎减半, 表现仍可比基线.

For global attention, CED treats the bottom $L / 2$ layers of the Transformer as the causal encoder. For the upper half layers (i. e., the decoder, $l > L / 2 )$ , the KV entries are not derived from their respective hidden states $H _ { l } .$ Instead, they are projected directly from the hidden state of the $( L / 2 )$ -th layer, $H _ { L / 2 }$ , using layer-dependent projection weights $\dot { ( W _ { l } ^ { K V } }$ and $W _ { l } ^ { Z } )$ :

where 𝐶 and 𝑍 represent the KV entries and their corresponding compression weights, respectively. This design allows CED to compute only the first half of the layers during the prefill phase, acquiring the upper-layer global KV cache with minimal computational cost.

全局注意力上, CED 把底部 $L/2$ 层当因果 encoder; 上半(decoder, $l > L/2$)的 KV 不从本层 $H_l$ 来, 而由第 $L/2$ 层隐状态 $H_{L/2}$ 经层相关投影 $W_l^{KV}$, $W_l^{Z}$ 得到(式 (1)). $C$, $Z$ 分别为 KV 条目及其压缩权重. 于是 prefill 只需算前半层, 就能以很低代价拿到上层 global KV.

For sliding window attention (SWA), CED maintains the conventional layer-wise computation across all layers. Specifically, for any layer 𝑙, the local keys and values are derived directly from the current layer’s hidden state $H _ { l } .$ This design effectively increases the computational depth of local KV generation. However, maintaining this layer-wise computation necessitates an SWA replay process. During the prefill phase, computing the SWA KV cache for the decoder requires processing an additional $n _ { \mathrm { w i n } } \times L / 2$ tokens (where $n _ { \mathrm { w i n } }$ denotes the window size). For multi-turn interactions with short prompts per turn, this computational overhead in the decoder becomes non-negligible. Fortunately, prior work (Chen et al., 2025) has shown that the actual effective receptive field of SWA is much smaller than the theoretical $n _ { \mathrm { w i n } } \times L / 2$ . Motivated by this observation, we introduce Decoder SWA Bounded Replay, which only prefills the last $n _ { \mathrm { w i n } }$ tokens of the prompt for the SWA computation, thereby significantly reducing the computational cost. Further details are provided in Section 3.2.2.

SWA 仍按层计算: 任意层 $l$ 的局部 K/V 直接来自 $H_l$, 抬高局部 KV 生成深度. 代价是需要 SWA 回放: prefill 时为 decoder 算 SWA KV, 额外要处理 $n_{\mathrm{win}} \times L/2$ 个 token. 多轮, 每轮短 prompt 时, 这段 decoder 开销不可忽视. 先前工作表明 SWA 有效感受野远小于理论 $n_{\mathrm{win}} \times L/2$, 于是提出 Decoder SWA Bounded Replay: SWA 只对 prompt 末 $n_{\mathrm{win}}$ 个 token 做 prefill, 显著省算力. 细节见 §3.2.2.

Overall, for a sequence length $N \gg n _ { \mathrm { w i n } } ,$ CED reduces the prefill complexity from $O ( N L )$ to $\bar { O ( N L / 2 + n _ { \mathrm { w i n } } \times L / 2 ) } \approx \bar { O ( N L / 2 ) }$ , effectively halving the overall computation.

当 $N \gg n_{\mathrm{win}}$ 时, prefill 复杂度从 $O(NL)$ 降到约 $O(NL/2 + n_{\mathrm{win}} \times L/2) \approx O(NL/2)$, 整体近乎减半.

2.3. Compressed Sparse Attention 2 (CSA2)

2.3. Compressed Sparse Attention 2(CSA2)

Serving long contexts requires controlling both KV cache storage and attention computation. These costs can be reduced along three multiplicative dimensions: the entry size, where GQA (Ainslie et al., 2023) reduces the number of KV heads and MLA (DeepSeek-AI, 2024) shares a small latent across heads; the sequence dimension, where every 𝑚 tokens are compressed into one entry, like CSA and HCA in DeepSeek-V4 (DeepSeek-AI, 2026b); and the layer dimension, where some layers reuse the caches (Brandon et al., 2024) and selections of other layers instead of keeping their own, or are replaced altogether by more efficient layers. Prior work has shown that compression along the layer dimension is effective: IndexCache (Bai et al., 2026) reuses Top-K indices across layers to cut indexer computation; YOIO (Sun et al., 2026b) computes the sparse routing once and shares it across all layers; and HySparse (Gao et al., 2026) lets sparse layers reuse the KV cache of dense layers. However, index reuse alone saves no main

服务长上下文, 要同时控 KV 存储与注意力计算. 成本可沿三个相乘维度压: 条目大小--GQA 减 KV 头数, MLA 跨头共享小潜变量; 序列维-- 每 $m$ 个 token 压成一条, 如 V4 的 CSA/HCA; 层维-- 部分层复用他层 cache/选择, 或换成更高效层. 层维压缩已被验证: IndexCache 跨层复用 Top-K 省 indexer; YOIO 稀疏路由算一次全网共享; HySparse 让稀疏层复用稠密层 KV. 但只复用索引省不了 main

Figure 4 | Three operating modes of CSA2. The modes differ in how they obtain main KV, indexer K, and Top-K indices. Green blocks indicate quantities computed in the current layer; yellow blocks indicate main KV and indexer K reused from the most recent Full Mode layer; while red blocks indicate Top-K indices reused from the most recent index-producing (Full or Reindex Mode) layer. All three modes compute main Q and SWA KV in the current layer.

图 4｜CSA2 三种工作模式: 获取 main KV, indexer K, Top-K 的方式不同. 绿块为本层计算; 黄块为复用最近 Full 层的 main KV 与 indexer K; 红块为复用最近产出索引(Full 或 Reindex)层的 Top-K. 三模式都在本层算 main Q 与 SWA KV.

KV storage, network-wide routing sharing limits performance, and hybrid designs still retain full attention layers; more importantly, none of these methods covers all three multiplicative dimensions.

KV 存储; 全网共享路由会限性能; 混合设计仍留全注意力层-- 更重要的是, 没有方法同时覆盖上述三维.

CSA2 exploits the three dimensions jointly: it shares main KV and indexer K across layers and allows layers to reuse Top-K indices, with cache sharing and index reuse decoupled. It combines these reuse strategies with a simplified compressor and a Hierarchical Sparse Indexer that narrows the search domain of subsequent indexing layers in the Decoder.

CSA2 联合吃三维: 跨层共享 main KV 与 indexer K, 并允许复用 Top-K; cache 共享与索引复用解耦. 再配简化压缩器, 以及缩小 decoder 后续索引层搜索域的分层稀疏 Indexer.

Similar to CSA, CSA2 includes a lightweight indexer that scores the main KV entries using indexer Q and indexer K and selects the Top-K entries for each query. Each Q attends to the selected entries together with the layer-local sliding-window KV (SWA KV). CSA2 also includes the uncompressed main KV setting as a special case with a compression ratio of 1. Meanwhile, CSA2 simplifies both the compressor and the indexer. In CSA, a compression ratio of 𝑚 produces each main KV entry from 2𝑚 original KV cache entries, with overlapping source entries for adjacent compressed entries. It also includes absolute positional embedding to encode the positions of these 2𝑚 entries during compression. CSA2 removes this overlap and absolute positional embedding. In addition, CSA2 obtains indexer K by projecting main KV entries, replacing CSA’s separate compression path from hidden states. Both designs simplify the implementation and increase the training efficiency.

与 CSA 类似, 轻量 indexer 用 indexer Q/K 给 main KV 打分并选 Top-K; 每个 Q 同时看选中条目与层内 SWA KV. 压缩比 1 时即未压缩 main KV. CSA2 简化压缩器与 indexer: CSA 在压缩比 $m$ 时每条 main KV 来自 $2m$ 条原始条目且相邻条目源重叠, 并加绝对位置嵌入; CSA2 去掉重叠与绝对位置嵌入, 且 indexer K 由 main KV 投影得到, 不再从隐状态另开压缩路径-- 实现更简, 训练更高效.

Sections 2.3.1 and 2.3.2 describe the cross-layer reuse strategies and the Hierarchical Sparse Indexer, respectively.

§2.3.1 / §2.3.2 分别讲跨层复用与分层稀疏 Indexer.

2.3.1. Cross-Layer KV and Index Reuse 跨层 KV 与索引复用

Each CSA2 layer is statically assigned one of three modes: Full, Reindex, or Reuse. In all three modes, the layer computes its own query and SWA KV and uses them together with the selected main KV entries to produce a new attention output. The modes differ in how they obtain main KV, indexer K, and Top-K indices. Figure 4 illustrates the three modes.

每层静态分配 Full / Reindex / Reuse 之一. 三模式都算本层 query 与 SWA KV, 并与选中的 main KV 一起出注意力; 差别只在 main KV, indexer K, Top-K 从哪来. 见图 4.

Full Mode. The layer computes its own main KV and indexer Q, projects indexer K from that main KV, and runs the indexer to produce fresh Top-K indices. It therefore executes the complete CSA2 computation path and has the same component responsibilities as a complete

Full. 本层算 main KV 与 indexer Q, 从 main KV 投影 indexer K, 跑 indexer 出新 Top-K-- 完整 CSA2 路径, 职责等同完整

Figure 5 | Hierarchical Sparse Indexer. Each square represents a position; green squares mark selected indices, and blue rectangles mark blocks selected based on their maximum indexer scores. The decoder’s first CSA2 layer in Full mode selects its own Top-512 indices and builds a shared candidate pool from the selected blocks for subsequent layers. CSA2 layers in Reindex mode then select their Top-512 indices from this pool.

图 5｜分层稀疏 Indexer. 每格一个位置; 绿格为选中索引, 蓝框为按块内最大 indexer 分数选出的块. Decoder 第一个 Full 的 CSA2 层选出自己的 Top-512, 再由选中块建共享候选池; 后续 Reindex 层从该池再选 Top-512.

CSA layer in DeepSeek-V4.

Reindex Mode. The layer reuses the most recent available main KV from a preceding layer together with its corresponding indexer K. The indexer computes its own query, rescores the reused keys, and produces fresh Top-K indices. This allows the sparse selection to change across layers while main KV and indexer K remain shared.

CSA 层(相对 V4).

Reindex. 复用前层最近可用的 main KV 与对应 indexer K; 本层 indexer 自算 query, 对复用的 key 重打分, 出新 Top-K-- 稀疏选择可跨层变, cache 仍共享.

Reuse Mode. The layer reuses the most recent available main KV and the latest Top-K indices computed against that main KV by a preceding layer in Full or Reindex Mode. It performs attention using this selection without computing indexer Q or evaluating index scores.

Reuse. 复用最近可用 main KV, 以及前层 Full/Reindex 针对该 KV 算出的最新 Top-K; 直接用该选择做注意力, 不算 indexer Q, 不打索引分.

Sharing main KV and indexer K reduces cache storage, while reusing Top-K indices avoids additional indexer computation. Reindex Mode preserves cache sharing while allowing the selected entries to change across layers. When CSA2 is combined with CED, the decoder layer assigned to Full Mode computes its own global KV from the hidden state of the (𝐿/2)-th layer, i. e. the last layer of the causal encoder. The Reindex and Reuse Modes are unchanged.

共享 main KV / indexer K 省存储, 复用 Top-K 省 indexer. Reindex 在共享 cache 的同时允许选择变化. 与 CED 结合时, decoder 的 Full 层从第 $L/2$ 层(encoder 末层)隐状态算自己的 global KV; Reindex / Reuse 不变.

2.3.2. Hierarchical Sparse Indexer 分层稀疏 Indexer

Cross-layer index reuse reduces the number of indexer evaluations, but the remaining indexers still score the full causally visible context. For extremely long contexts, this cost remains a major computational bottleneck. Prior work introduced indexer sparsity by scoring and pruning pooled block representations before token-level indexing (Xu et al., 2026b). We find that in the decoder, information from shallower indexers can naturally be used to restrict the candidates considered by deeper indexers without adding any extra state. We therefore introduce the Hierarchical Sparse Indexer, which is used only in the decoder of CED to reduce this repeated scoring during decode. For each query, the first layer assigned to Full Mode constructs a candidate pool that later re-indexing layers use as their search domain. For a fixed candidatepool size, this changes the per-query cost of deeper indexers from linear in context length to

跨层索引复用减少了 indexer 评估次数, 但剩下的 indexer 仍要对全部因果可见上下文打分; 极长上下文时这仍是大计算瓶颈. 先前工作用池化块表示先打分剪枝再做 token 级索引. 我们发现 decoder 里浅层 indexer 的信息可自然限制深层候选, 且不加额外状态, 于是引入仅用于 CED decoder 的分层稀疏 Indexer, 压 decode 期重复打分. 每个 query: 第一个 Full 层建候选池, 后续重索引层只在池内搜. 候选池大小固定时, 深层 indexer 每 query 代价从随上下文线性变成

constant. The mechanism is training-aware and introduced in post-training: the candidate restriction is applied identically during training and inference, so deeper indexers are optimized under the same search domain they use at inference. Figure 5 illustrates this process.

常数. 机制训练期感知, 在后训练引入: 训练与推理用同一候选限制, 深层 indexer 在推理同款搜索域上优化. 见图 5.

This first Full Mode layer scores all causally visible main KV positions and produces the Top-K indices for its own attention. It also performs blockwise candidate selection: each block is assigned the maximum index score among its positions, and the blocks with the highest scores are selected. It then collects the positions covered by the selected blocks into a candidate pool larger than the final Top-K set. For example, selecting 2, 048 blocks with 8 positions each yields 16, 384 candidate positions. This pool defines where later indexers search; the final Top-K selection determines which main KV entries each layer reads.

该 Full 层对全部因果可见 main KV 打分, 产出本层注意力用的 Top-K; 同时做块级候选: 每块取块内最大索引分, 选高分块, 再把块内位置收成大于最终 Top-K 的候选池. 例: 选 2048 块×每块 8 位置 → 16384 候选. 池定义后续搜索范围; 最终 Top-K 决定每层读哪些 main KV.

Subsequent layers in Reindex Mode score only the candidate positions for the corresponding query and select their own Top-K entries within that pool. Layers in Reuse Mode perform no new indexing and use the latest Top-K indices computed against the main KV they reuse. Thus, the candidate pool is shared across indexing layers, while their final selections can differ.

后续 Reindex 层只对对应 query 的候选位置打分, 在池内自选 Top-K; Reuse 层不再索引, 用其复用 main KV 上最新 Top-K. 候选池共享, 最终选择可不同.

For a fixed candidate-pool size, the number of positions scored per query by each subsequent indexer is bounded independently of context length. The first Full Mode layer still scans the entire causally visible range. Hierarchical indexing therefore reduces the cost of later indexer evaluations while retaining the initial full-range pass.

候选池固定时, 后续 indexer 每 query 打分位置数与上下文长度无关; 首个 Full 层仍扫全因果可见范围-- 分层索引压后期代价, 同时保留首轮全范围扫描.

2.4. Efficient Architectural Extensions 高效架构扩展

2.4.1. Single-Pass mHC

2.4.1. Single-Pass mHC

In DeepSeek-V4, we introduced mHC (Xie et al., 2026), which maintains 𝑛 residual streams between adjacent Transformer blocks. For each token, we denote these streams by $X _ { l } \in \mathbb { R } ^ { n \times d }$ where 𝑙 is the block index and 𝑑 is the hidden dimension. The streams are updated as follows:

where $A _ { l } \in \mathbb { R } ^ { 1 \times n } , C _ { l } \in \mathbb { R } ^ { n \times 1 }$ and $B _ { l } \in \mathbb { R } ^ { n \times n }$ are token-wise coefficients predicted from 𝑋𝑙. The coefficient predictor H includes normalization and projection.

V4 引入 mHC: 相邻 Transformer 块之间维护 $n$ 条残差流. 每 token 记 $X_l \in \mathbb{R}^{n \times d}$, 更新见式 (2). $A_l$, $B_l$, $C_l$ 为从 $X_l$ 预测的 token 级系数; 预测器 H 含归一化与投影.

Ideally, the residual transformation between two blocks is a single map from $( X _ { l - 1 } , Y _ { l - 1 } )$ to $( X _ { l } , \hat { X } _ { l } )$ , where $\hat { X } _ { l } = A _ { l } X _ { l }$ is the current block input and $Y _ { l - 1 } = \mathcal { F } _ { l - 1 } ( \hat { X } _ { l - 1 } )$ is the previous block output. Such a map requires (𝑛 + 1)𝑑 reads and $( n + 1 ) d$ writes, giving a lower bound of $( 2 n + 2 ) d$ on activation memory traffic. In practice, DeepSeek-V4 uses a multi-pass implementation of (2), with three kernels that execute sequentially due to data dependencies:

The three kernels read $( n + 1 ) d ,$ , 𝑛𝑑 and 𝑛𝑑 values respectively and write $( n + 1 ) d$ in total. Including the pre-norm in $\mathcal { F } _ { l } ,$ the total activation memory traffic is $( 4 n + 4 ) d ,$ twice the lower bound.

理想上两块之间残差变换是一次映射: $(X_{l-1}, Y_{l-1}) \to (X_l, \hat{X}_l)$, 读写下界 $(2n+2)d$. 实际 V4 用多趟实现(式 (3)–(5) 三核串行), 计入 $\mathcal{F}_l$ 的 pre-norm 后激活流量 $(4n+4)d$, 是下界的两倍.

In H, the normalization weights are folded into the projection weights offline, and the RMS division is applied after the projection. Two of the three stages can therefore share one traversal of the residual: the residual update does not require a reduction across the hidden dimension, so each tile of $X _ { l }$ can be computed and immediately used to accumulate the projection outputs

在 H 里, 归一化权重离线折进投影, RMS 除法放投影后. 三阶段里两段可共享一次残差遍历: 残差更新无需跨隐维归约, 每个 $X_l$ tile 算完即可立刻累加投影输出

and the sum of squares needed to compute the RMS. Input mixing cannot be fused into this pass because $A _ { l }$ is not available until the reduction over all hidden tiles is complete. It therefore requires a second read of $X _ { l } .$ This second pass can also incorporate input pre-norm. This twopass implementation would require (3𝑛 + 2)𝑑 activation reads and writes in total, one additional read of $X _ { l }$ compared with the lower bound.

与 RMS 所需平方和. 输入混合因 $A_l$ 要等全部隐维 tile 归约完才可用, 无法并进此趟, 需再读一次 $X_l$(可顺带 pre-norm). 两趟合计 $(3n+2)d$, 比下界多一次 $X_l$ 读.

We therefore introduce Single-Pass mHC, which shifts the input-mixing coefficients by one block, i. e. every block consumes the mixing coefficients produced by the previous one, so that the above dependency disappears:

Input mixing now uses $A _ { l - 1 }$ instead of $A _ { l } ,$ so it no longer depends on the coefficients computed from $X _ { l } .$ Each tile of 𝑋_𝑙 can therefore be used immediately for both input mixing and coefficient prediction, without waiting for the full reduction. Empirically, this shift incurs negligible performance degradation.

于是提出 Single-Pass mHC: 把输入混合系数错开一块(式 (6))-- 每块用上一块产出的混合系数, 依赖消失. 输入混合改用 $A_{l-1}$, 每个 $X_l$ tile 可立刻同时做混合与系数预测. 经验上该错位几乎不伤性能.

For pre-training, we keep the existing multi-kernel implementation, since the shift only changes which mixing coefficients each block applies. For deployment, we fuse residual update, input mixing, and coefficient prediction into a single kernel, Mega-mHC. The kernel implements mHC with (3𝑛 + 2)𝑑 activation reads and writes and Single-Pass mHC with $( 2 n + 2 ) d$ activation reads and writes. Mega-mHC processes $X _ { l }$ in tiles along the hidden dimension. Each tile is used to compute the mixed input and to accumulate the quantities needed to predict $( A _ { l } , B _ { l } , C _ { l } )$ for the next block. The kernel also incorporates input pre-norm and FP8 conversion. The residual is thereby read once and written once, attaining the (𝑛 + 1)𝑑 reads and (𝑛 + 1)𝑑 writes of the ideal map and halving the activation memory traffic of our original implementation.

预训练仍用多核实现(错位只改每块用哪套系数). 部署上把残差更新, 输入混合, 系数预测融成 Mega-mHC: 实现原 mHC 时流量 $(3n+2)d$, Single-Pass 时 $(2n+2)d$. 沿隐维分 tile 处理 $X_l$, 每 tile 同时算混合输入并累加下一块所需 $(A_l, B_l, C_l)$, 并含 pre-norm 与 FP8 转换. 残差一读一写, 达到理想映射的 $(n+1)d$ 读写, 相对原实现流量减半.

2.4.2. Engram

2.4.2. Engram

We augment DeepSeek-V4.1-Flash with Engram (Cheng et al., 2026c), the conditional memory module introduced in our previous work to decouple memorization from computation. We follow the original Engram design-tokenizer compression, multi-head hashing, context-aware gating, and multi-branch integration-with two modifications. First, we omit the short causal convolution because its performance gains do not justify the added complexity in our inference stack. Second, we optimize the Engram embedding with momentum-based update followed by Sinkhorn balancing, as detailed in Section 2.5.

接入 Engram 条件记忆模块, 把记忆与计算解耦. 沿用原设计(分词器压缩, 多头哈希, 上下文门控, 多分支融合), 两处改动: 去掉短因果卷积(收益不抵推理栈复杂度); Engram 嵌入改动量更新 + Sinkhorn 均衡(见 §2.5).

We allocate 196B Engram parameters evenly across two modules. Each module uses 𝑁-gram orders {2, 3, 4}, with 8 hash heads and a total embedding dimension of 2048 per order. Each head indexes a table of approximately 16M entries, with table sizes chosen to be distinct primes. Both the embedding tables and the key/value projections use FP8 precision. The modules are placed at layers 1 and 14 (zero-indexed) to balance memory usage across training pipeline stages. During inference, deterministic addressing enables embeddings to be prefetched from host memory via background RDMA transfers, with prefetching for the first module overlapping computation in the first Transformer block. Further implementation details for Engram training and inference are discussed in Section 3.1.3.

196B Engram 参均分两个模块; 每模块 N-gram 阶 {2, 3, 4}, 8 个哈希头, 每阶嵌入总维 2048; 每头约 16M 表项, 表大小取互异素数. 表与 K/V 投影均 FP8. 模块放在第 1, 14 层(从 0 计), 平衡流水级显存. 推理时确定性寻址, 可经后台 RDMA 从主机内存预取; 第一模块预取与第一 Transformer 块计算重叠. 训练/推理细节见 §3.1.3.

2.4.3. DSpark

2.4.3. DSpark

We equip DeepSeek-V4.1-Flash with DSpark (Cheng et al., 2026a), a speculative decoding module that combines semi-autoregressive drafting with confidence-scheduled verification.

配备 DSpark: 半自回归起草 + 按置信度调度校验的投机解码模块.

The drafter comprises three Transformer blocks with a sliding attention window of 128 tokens. A single forward pass through these blocks computes base logits for five draft positions in parallel, while a lightweight Markov head models dependencies among the draft tokens. A confidence head predicts per-position conditional acceptance probabilities, which are used to estimate prefix survival probabilities. The scheduler combines these estimates with profiled engine throughput curves to dynamically select the verification length for each request, aiming to maximize expected system-wide token throughput under the current system load.

起草器三层 Transformer, 滑动注意力窗 128. 一次前向并行出五个草稿位的 base logits, 轻量 Markov 头建模草稿 token 依赖; 置信度头预测每位置条件接受概率, 用以估计前缀存活概率. 调度器结合引擎吞吐曲线, 为每请求动态选校验长度, 目标是在当前负载下最大化系统期望 token 吞吐.

Unlike the MTP module in DeepSeek-V3 (DeepSeek-AI, 2024), which is trained jointly with the backbone throughout pre-training, DSpark is introduced in a dedicated stage after pre-training. In this stage, we train only DSpark while keeping the backbone frozen. During post-training, we continue to train DSpark alongside the backbone, without propagating gradients from the DSpark objective into the backbone. This keeps DSpark aligned with the evolving policy, enabling it to accelerate both online serving and rollout generation for RL and OPD.

与 V3 全程与骨干共训的 MTP 不同, DSpark 在预训练后单独阶段引入: 只训 DSpark, 冻骨干. 后训练期继续与骨干并行训, 但不把 DSpark 目标梯度回传到骨干-- 让它跟上演化中的策略, 同时加速在线服务与 RL/OPD 的 rollout.

2.4.4. FP4 Main KV Cache

2.4.4. FP4 主 KV Cache

Long-context agent workloads require large per-request KV caches, increasing serving costs. DeepSeek-V4 already uses quantization-aware training (QAT) (Jacob et al., 2018) for FP4 indexer queries and keys, accelerating index computation and reducing the indexer cache size. We adopt the OCP-standard MXFP4 format (Rouhani et al., 2023) to support as many hardware platforms as possible, despite the higher accuracy of alternative formats in our experiments. We now extend QAT to the main KV cache, where FP4 reduces storage rather than accelerates matrix multiplication. Dequantizing cached values before attention allows us to use a more accurate format without requiring native matrix-multiplication support for that format, preserving compatibility across hardware platforms.

长上下文 Agent 每请求 KV 很大, 抬服务成本. V4 已对 indexer Q/K 做 FP4 QAT; 此处采用 OCP 标准 MXFP4 以覆盖更多硬件(尽管实验里别的格式更准). 现把 QAT 扩到主 KV: FP4 主要省存储而非加速矩阵乘. 注意力前反量化, 可用更准格式且不要求硬件原生该格式的 matmul, 兼顾兼容.

Among the approximately four-bit formats evaluated, we select E2M1 with one E4M3 scale per 16 channels, following NVFP4 (Alvarez et al., 2025) but omitting its second-level global scale to balance accuracy and simplicity. Omitting this scale leaves ample dynamic range for the main KV cache: the format supports magnitudes up to 448 × 6 = 2688, far above the cache’s magnitude bound. In DeepSeek-V4.1-Flash, the largest trained RMSNorm weight magnitude is approximately 1. After RMS normalization, the L2 norm of the 512-channel KV latent is at most approximately √512. RoPE preserves this norm, so the maximum absolute value across channels after rotation is also bounded by approximately √512 ≈ 22.6. Besides, the maximum magnitude observed during training is around 10. Therefore, omitting the global scale causes no measurable decrease in accuracy and simplifies the cache layout.

约四比特格式里选 E2M1, 每 16 通道一个 E4M3 scale(跟 NVFP4, 但去掉二级全局 scale). 去掉后动态范围仍够: 最大幅值可达 448×6=2688, 远高于 cache 量级. 本模型训练到的 RMSNorm 权重幅值最大约 1; RMS 后 512 通道 KV 潜变量 L2 至多约 √512; RoPE 保范数, 旋转后通道最大绝对值亦约 √512≈22.6; 训练观测最大约 10. 故去全局 scale 无明显掉点, 布局更简.

To enable FP4 main KV cache storage in DeepSeek-V4.1-Flash, we introduce QAT during post-training. The non-RoPE and RoPE components use the same quantization format. We quantize the cache after RoPE: quantizing before RoPE yields only a marginal accuracy improvement in our experiments and would introduce additional overhead during decoding. We retain FP8 for the SWA KV cache due to its sensitivity to quantization. Compared with the FP8 main KV cache in DeepSeek-V4, this format nearly halves the storage footprint, both in HBM and when offloaded to SSD.

后训练引入 QAT 以启用 FP4 主 KV. 非 RoPE 与 RoPE 分量同格式; 在 RoPE 之后量化(之前量化仅边际收益且解码更贵). SWA KV 对量化敏感, 仍留 FP8. 相对 V4 的 FP8 主 KV, HBM 与卸到 SSD 时存储近乎减半.

2.5. Optimization 优化

Building upon the optimization configuration used in DeepSeek-V4, we make some new modifications to better align with the architectural design.

在 V4 优化配置上做若干对齐新架构的修改.

First, we use head-wise Muon, where Query weights are split by head before applying the

其一, head-wise Muon: Query 权重先按头切再做

Muon update. Here, we briefly discuss the motivation for such a design. By viewing Muon as a preconditioned gradient descent, vanilla Muon uses one preconditioner for all heads, whereas head-wise Muon provides different preconditioners for different heads. This design can better handle the heterogeneity across attention heads (Zhang et al., 2024; Zhang, 2026, Section 3). As a result, we observe that head-wise Muon outperforms vanilla Muon. The empirical advantage of head-wise Muon is also validated in GLM 5 (Zeng et al., 2026) and Kimi-K3 (Team et al., 2026a).

Muon 更新. 把 Muon 看成预条件梯度下降: 原版所有头共用一个预条件, head-wise 则每头不同, 更能处理注意力头异质性; 实验上优于原版, GLM-5, Kimi-K3 也验证了这点.

Second, applying Adam to the newly introduced Engram parameters substantially increases the optimizer-state memory footprint. To reduce memory usage during training, we instead optimize the Engram embedding tables, token embedding, and prediction head using a momentumbased update followed by Sinkhorn balancing. Sinkhorn balancing has previously been applied to linear-layer weight matrices in SinkGD (Scetbon et al., 2025); here, we extend it to these large parameter matrices. Like Muon, this approach requires only a momentum buffer while empirically outperforming Adam.

其二, 对 Engram 用 Adam 会大幅抬优化器状态显存. 于是 Engram 表, token 嵌入与预测头改用动量更新 + Sinkhorn 均衡(SinkGD 曾用于线性层, 这里扩到大矩阵). 与 Muon 一样只需动量缓冲, 经验上优于 Adam.

Basic Configurations. We retain AdamW (Loshchilov and Hutter, 2019) for normalizationlayer weights and other non-matrix parameters, including biases and scaling factors. We use Muon (Jordan et al., 2024) for the weight matrices of linear transformations in the languagemodel backbone, the Engram projection layers, and the vision-language projector. We use head-wise Muon for Query and Key weights. We apply decoupled weight decay and Nesterov momentum to Muon (Nesterov, 1983; Liu et al., 2025); normalization-layer weights are also subject to weight decay, whereas biases and scaling factors are not. The Sinkhorn-balanced update also uses Nesterov momentum but does not apply weight decay. During pre-training, we keep the vision encoder frozen until the learning-rate decay stage, while its final normalization layer and the vision–language projector remain trainable. At the onset of learning-rate decay, we unfreeze the vision encoder and optimize it jointly with the LLM with a smaller learning rate.

基本配置. 归一化层权重与其他非矩阵参(偏置, 缩放)仍 AdamW; 语言骨干线性层, Engram 投影, 视觉–语言投影用 Muon; Q/K 用 head-wise Muon. Muon 解耦 weight decay + Nesterov; 归一化层也有 weight decay, 偏置/缩放没有. Sinkhorn 更新亦 Nesterov, 但无 weight decay. 预训练中视觉编码器冻到学习率衰减阶段, 末层归一化与投影可训; 衰减开始后解冻视觉编码器, 与 LLM 联合训但用更小学习率.

Sinkhorn-Balanced Updates for Engram / Embedding / Prediction Head. The complete procedure is summarized in Algorithm 1. At a high level, it follows the same workflow as Muon, with Sinkhorn balancing taking the place of Newton–Schulz orthogonalization. We denote the larger matrix dimension by $m ,$ which corresponds to the vocabulary size for embedding tables and prediction heads, and denote the hidden dimension by 𝑛.

Given the Nesterov momentum update $\widehat { G } _ { t , i }$ Sinkhorn balancing finds diagonal scaling matrices $D _ { r }$ and $D _ { c }$ such that

Thus, the procedure approximately equalizes the row-wise and column-wise RMS of the update matrix. Here, one row corresponds to one token index or n-gram identity; and one column encodes one hidden feature. Sinkhorn balancing exploits this token–feature structure by normalizing along both rows and columns. For numerical stability, rows satisfying $\rho _ { i } \leqslant \tau \bar { \rho }$ are masked. The factor $\sqrt { n }$ converts unit row $\ell _ { 2 }$ norm into unit row-wise RMS. Separately, we adjust the effective learning rate as $\widetilde { \eta } _ { t } = \gamma \eta _ { t }$ to match the update magnitude of Adam. We set $\gamma = 0 . 1 8 ,$ which is close to the factor 0.2 used in Moonlight (Liu et al., 2025).

Engram / 嵌入 / 预测头的 Sinkhorn 均衡更新. 算法 1; 流程似 Muon, 用 Sinkhorn 代替 Newton–Schulz. 大维记 $m$(词表/表大小), 隐维记 $n$. 对 Nesterov 动量更新做行列缩放, 使更新矩阵行/列 RMS 近似为 1(式 (7)). 行对应 token/n-gram, 列对应隐特征. $\rho_i \leqslant \tau\bar{\rho}$ 的行掩掉; $\sqrt{n}$ 把单位行 $\ell_2$ 转成单位行 RMS. 有效学习率 $\widetilde{\eta}_t=\gamma\eta_t$, $\gamma=0.18$(接近 Moonlight 的 0.2).

More broadly, Sinkhorn balancing is closely related to optimizers that exploit matrix or tensor axis structure (Shazeer and Stern, 2018; Zhang et al., 2025a; Wen et al., 2025; Glentis et al., 2025; Deng et al., 2026; Yuan et al., 2026; Xu et al., 2026a). For example, Adafactor (Shazeer and

更广地说, Sinkhorn 均衡与利用矩阵/张量轴结构的优化器一族相近(Adafactor 等). 例如 Adafactor(Shazeer and

算法 1｜动量更新 + Sinkhorn 均衡(流程与源文伪代码一致: Nesterov 动量 → 掩近零行 → 奇偶步交替行/列归一化 → $\sqrt{n}$ 转 RMS → 校正学习率后更新).

Stern, 2018) conducts row- and column-wise normalization in a different manner, and Adammini (Zhang et al., 2025a) uses an alternative row-wise normalization for embedding tables and prediction head. These normalization strategies may differ in optimization performance and communication overhead. We leave more detailed investigation as a future direction.

Stern, 2018)以不同方式做行列归一化; Adam-mini 对嵌入与预测头用另一种行归一化. 性能与通信开销可能各异, 细究留待未来.

3. General Infrastructures 通用基础设施

3.1. Training Infrastructure 训练基础设施

3.1.1. Multimodal Training Infrastructure 多模态训练基础设施

Communication-Computation Overlap in Contrastive Learning. The vision encoder is first optimized with a contrastive objective before being fine-tuned with a generative next-token prediction loss. In the contrastive phase, the loss is computed over a full batch of text and vision pairs, so the features of both modalities must be all-gathered across data-parallel ranks, incurring substantial communication. Because the gradient of the text features depends only on the gathered visual features-and, symmetrically, the gradient of the visual features depends only on the gathered text features-each all-gather can be overlapped with the forward or backward pass instead of stalling the pipeline:

where 𝑉 and 𝑇 denote the visual and text features, $( A \parallel C )$ denotes the overlap of computation 𝐴 with communication $C ,$ and ∇ denotes the gradient computation. In this schedule, the visual

对比学习中的通信–计算重叠. 视觉编码器先对比目标, 再生成式下一词微调. 对比阶段要对整批图文对算损失, 两模态特征需跨 DP 做 all-gather. 因文本特征梯度只依赖汇齐的视觉特征(对称地, 视觉梯度只依赖汇齐的文本特征), all-gather 可与前/反向重叠, 不必堵流水(调度见源文公式). 此调度下, 视觉

features are gathered during the text forward pass and the text features during the text backward pass, so that both all-gathers are hidden entirely behind useful computation.

特征在文本前向时汇齐, 文本特征在文本反向时汇齐, 两次 all-gather 都藏在有用计算背后.

End-to-End Parallelism. To handle the model and data heterogeneity between the vision encoder and the LLM (Zhang et al., 2025b), we adopt the disaggregated encoder design used in recent training systems (Team et al., 2025, 2026b). The vision encoder is replicated outside the LLM parameter tree, and each training step is divided into three phases: vision encoder forward, LLM forward/backward, and vision encoder backward. This separation prevents interference between vision encoder and LLM computation. Load-balanced vision processing is confined to the first and last phases, while the LLM phase remains free of vision computation and preserves the parallel strategy of text-only training.

端到端并行. 为处理视觉编码器与 LLM 的模型/数据异质性, 采用解耦编码器: 视觉编码器复制在 LLM 参数树外, 每步分三段-- 视觉前向, LLM 前/反向, 视觉反向. 均衡视觉处理只落在首尾段, LLM 段无视觉计算, 保留纯文本并行策略.

Long-Sequence Multimodal Training Optimization. DeepSeek-V4.1-Flash is trained on sequences of up to one million tokens, with a substantial share of training occurring at ultra-long sequence lengths in both the pre-training and post-training stages. At these sequence lengths, multimodal samples create heavy I/O, CPU, and memory bottlenecks.

长序列多模态训练优化. 序列最长 100 万 token, 预训练与后训练大量处在超长序列; 多模态样本会重压 I/O, CPU 与内存.

• Balanced image sharding. During pre-training, a single ultra-long, image-dense sequence can exhaust one host’s $\mathrm { I } / \mathrm { O } ,$ CPU, and memory during loading, so the images of each sequence are sharded across the CP ranks with load balancing, and each image is loaded exactly once. With images read once, loading stays hidden behind compute whenever

where 𝑁 is the token count, $\rho$ the raw bytes per token, 𝐶 the per-token compute, and $B _ { \mathrm { I O } , }$ $B _ { \mathrm { G P U } }$ the file-system and GPU bandwidths. Since 𝑁 cancels, the criterion involves only pertoken quantities (𝜌 and 𝐶), independent of sequence length and cluster size; $\rho$ is set by the vision-module configuration (e. g., the resolution cap or the spatial downsample). Storage throughput therefore becomes a bottleneck only for small models with low per-token compute, as in ablations, while production-scale models remain compute-bound.

• 均衡图像分片. 超长, 图密序列会在加载时榨干单机 I/O/CPU/内存, 故跨 CP rank 均衡分片, 每图只加载一次. 满足上式时加载可藏在计算后; 准则只依赖每 token 量($\rho$, $C$), 与序列长与集群规模无关. 生产规模模型仍偏计算受限, 存储吞吐瓶颈多见于消融用的小模型.

• Incremental image transfer. Besides the balanced sharding above, the reinforcementlearning rollout transfers images to the inference engine only incrementally, and caches the engine’s CPU-side decoding and preprocessing outputs on a distributed file system for reuse across rollouts and subsequent training.

• 增量图像传输. RL rollout 向推理引擎增量传图, 并把 CPU 侧解码与预处理结果缓存在分布式文件系统, 供跨 rollout 与后续训练复用.

3.1.2. Attention Sharing Training for CSA2 CSA2 的注意力共享训练

In Section 2.3, we introduce CSA2, an attention sharing method with three modes, some of which involve sharing one or more of the main KV, the indexer K, and the Top-K indices across multiple layers. Supporting CSA2 in large-scale distributed training requires additional coordination beyond the attention computation itself. In particular, layers that share attention components may be placed on different pipeline stages, making direct module reuse incompatible with conventional stage-local execution. Therefore, we adopt several designs to support CSA2 training.

CSA2 三模式会跨层共享 main KV / indexer K / Top-K 之一或多者. 大规模分布式训练里, 共享组件可能落在不同流水级, 直接模块复用与「级内执行」不兼容, 故另加设计.

Shadow indexers address this issue by placing a lightweight executable replica on each participating stage while retaining a single logical owner for the shared parameters. The owner remains responsible for optimization and checkpointing, whereas parameter synchronization

Shadow indexer: 参与级上放轻量可执行副本, 共享参数仍只有一个逻辑所有者(负责优化与 checkpoint); 参数同步

and gradient aggregation keep the shadow replicas consistent throughout training. This design preserves the original model semantics without requiring the pipeline scheduler to treat shared layers as a special execution unit.

与梯度聚合保持副本一致. 不改流水调度语义, 也不必把共享层当成特殊执行单元.

Pipeline payload extensions provide the intermediate representations and sparse routing information required by downstream consumers when the source and consumer layers cross a pipeline boundary. These states are incorporated into the existing point-to-point communication path and partitioned consistently with context parallelism, avoiding unnecessary replication while maintaining the corresponding gradient flow.

流水 payload 扩展: 源与消费层跨流水边界时, 把中间表示与稀疏路由信息塞进既有点对点通信, 并与上下文并行一致切分, 避免多余复制且保住梯度流.

Micro-batch-level shared-state management tracks the states associated with concurrently active pipeline micro-batches and coordinates their lifetimes across forward execution, activation recomputation, and backward propagation. Shared states are retained until their final consumer has completed and are then released promptly to limit additional memory overhead. The same runtime abstraction also handles stage placement and source–consumer relationships, allowing the attention implementation to access shared states without depending on the physical pipeline layout.

微批级共享状态管理: 跟踪并发活跃微批相关状态, 协调前向, 激活重算与反向的生命周期; 最终消费者完成后立即释放. 同一运行时抽象处理级放置与源–消费关系, 注意力实现不依赖物理流水布局.

Together with lightweight adaptations to the optimizer, checkpointing, warm-up, and computation-graph tracing workflows, these mechanisms enable CSA2 to operate transparently under the existing distributed training interface and pipeline schedules.

再加对优化器, checkpoint, warmup, 计算图追踪的轻量适配, CSA2 可在既有分布式训练接口与流水调度下透明运行.

3.1.3. Engram

3.1.3. Engram

Engram embedding tables are partitioned by row across dedicated process groups of engram parallel size. The group size controls the trade-off between per-device memory usage and the communication scope of embedding lookups. Optimizer states are further sharded across replicas of each table partition. Engram lookup indices depend solely on the input token sequence. Embedding prefetch is therefore initiated for the entire local batch before each pipeline stage begins processing microbatches for the current training step, minimizing interference with pipeline execution. Embedding gradients are buffered during backward and returned to their owning ranks after the backbone backward pass. For efficient integration with multimodal training, embedding prefetch and gradient transfers are scheduled to overlap with the vision encoder’s forward and backward computation. Embeddings are stored and fetched in FP8, with the retrieved values and scaling factors passed directly to the following GEMM. For Engram table updates, Sinkhorn normalization maintains row and column scaling vectors across iterations to avoid repeated writes of the full normalized matrix. Row normalization and the accumulation of partial column statistics are fused into a single kernel to further reduce memory traffic. During RL rollouts, Engram embedding tables remain resident in GPU memory. This placement reduces host memory pressure and helps avoid out-of-memory failures caused by host memory fragmentation.

Engram 表按行切到专用 engram-parallel 进程组; 组大小权衡每卡显存与查找通信范围. 优化器状态再按表分区副本切分. 查找索引只依赖输入 token, 故每步流水级开微批前就对本地整批预取嵌入. 反向缓冲梯度, 骨干反向后再回所有者. 与多模态训练结合时, 预取与梯度传输与视觉前/反向重叠. 嵌入 FP8 存取, 值与 scale 直接进后续 GEMM. 表更新用 Sinkhorn 跨迭代维护行列缩放向量, 避免反复写整表; 行归一化与部分列统计累加融成单核. RL rollout 期表常驻 GPU, 减轻主机内存压力与碎片 OOM.

3.2. Inference System 推理系统

DeepSeek-V4.1-Flash is designed with inference efficiency as a first-class concern. Although its architecture is conceptually complex, the resulting inference kernel flow is remarkably concise. Through reasonable kernel fusion, we encapsulate the intricate operations and keep hardware resources fully pipelined inside a small number of fused kernels-including the fused-RoPEattention-RoPE-cast kernel in FlashMLA (Li and Liu, 2025), the Mega-Gate, Mega-mHC, and Mega-MoE kernels in DeepGEMM (Zhao et al., 2025), the kernels in TileKernels (Wang et al., 2026a), and the TopK kernel in DeepSelect (Qian et al., 2026). As a result, the vast majority of

推理效率是一等公民. 架构概念虽复杂, 核流却很干净: 合理融合后, 复杂操作收进少数融合核(FlashMLA 的 fused-RoPE-attention-RoPE-cast, DeepGEMM 的 Mega-Gate / Mega-mHC / Mega-MoE, TileKernels, DeepSelect TopK 等). 结果是绝大多数

Transformer layers-those whose CSA2 operates in Reuse Mode-execute with only 15 kernels during prefill and 11 during decode, thereby achieving both high-throughput and low-latency inference.

Reuse 模式的 Transformer 层: prefill 仅 15 核, decode 仅 11 核, 高吞吐与低延迟兼得.

At the deployment level, we adopt Encoder–Prefill–Decode (EPD) disaggregation, enabling vision encoding, prefill, and decoding to scale independently and overlap in execution.

部署采用 Encoder–Prefill–Decode(EPD) 解耦: 视觉编码, prefill, decode 可独立扩展并重叠执行.

3.2.1. Persistent KV Cache Management Persistent KV Cache 管理

Under identical workloads, V4.1 reduces the persistent KV cache footprint to $1 / 8$ of that of V4. Two multiplicative factors account for this reduction: the persistent KV cache no longer stores SWA KV, which almost halves its size, and the global KV retained in it is further compressed to 1/4 of V4’s footprint through architectural and precision optimizations.

同负载下 V4.1 把 persistent KV 压到 V4 的 $1/8$. 两个相乘因子: 不再存 SWA KV(近乎减半), 保留的 global KV 经架构与精度再压到 V4 的 1/4.

In the V4 deployment, SWA KV accounts for nearly half of the persistent KV cache capacity. Within this cache, global KV and SWA KV are managed independently, governed by an LRU eviction policy. Global KV is stored in its entirety, and upon a hit, the complete prefix is reused. In contrast, SWA KV is cached at two specific points-the end of the prompt and the end of the output-to facilitate regeneration and multi-turn sessions; a hit allows computation to resume from that cached position. To maintain a high hit rate, we configured a sufficiently large persistent KV cache on SSD such that, under typical workloads, both types of KV remain resident for over 72 hours. Despite retaining only $n _ { \mathrm { w i n } }$ KV entries at designated positions, the uncompressed SWA KV cache still incurs a substantial storage overhead, especially in multi-turn conversations with short turns.

V4 部署里 SWA KV 约占 persistent 一半; global 与 SWA 独立管理, LRU 淘汰. Global 整段存, 命中复用完整前缀; SWA 只缓在 prompt 末与输出末两处, 命中可从该点续算. 为保命中率, SSD 上 persistent 够大, 典型负载下两类 KV 常驻超 72 小时. 即便指定位置只留 $n_{\mathrm{win}}$ 条, 未压缩 SWA 仍贵-- 短轮多轮对话尤其明显.

Persistently storing SWA KV is both costly and ineffective, because its access pattern does not match the persistent KV cache’s long retention policy. Unlike global KV, which exhibits long-tail reuse, SWA KV is reused only within a narrow, minute-scale window inside an active session and becomes dead once the session ends or the next turn begins. The V4 technical report proposed Zero SWA Caching, which avoids the storage overhead by recomputing missing SWA KV. Exact recovery, however, requires a full forward pass over $L \times n _ { \mathrm { w i n } }$ tokens, whose cost proved prohibitive in production deployments.

持久存 SWA 又贵又不对路: 它不像 global 有长尾复用, 只在活跃会话内分钟级窗口有用, 会话结束或下一轮即死. V4 报告提过 Zero SWA Caching(缺则重算), 但精确恢复要对 $L\times n_{\mathrm{win}}$ token 做完整前向, 生产上过贵.

V4.1 therefore revises persistent KV cache management as follows: V4.1 因此改写 persistent 管理如下:

1. SWA KV is no longer cached in the persistent KV cache and instead stored in a distributed memory pool provisioned from 10% of the host DRAM on each machine. Although this pool is far smaller in aggregate capacity, its short TTL (only minutes) allows expired entries to be recycled immediately for new sessions; under real-world workloads, this high turnover suffices to serve the vast majority of concurrent active sessions. Global KV remains in the persistent KV cache with a guaranteed lifetime of at least 72 hours.
2. SWA KV 不再进 persistent, 改放每机主机 DRAM 约 10% 划出的分布式内存池. 总容量小, 但 TTL 仅分钟级, 过期立即回收; 真实负载下高周转可服务绝大多数并发活跃会话. Global KV 仍在 persistent, 保证寿命至少 72 小时.
3. Evicting SWA KV inevitably causes misses, which stay affordable thanks to a lightweight fallback, Encoder SWA Bounded Replay (detailed in Section 3.2.2). For the inevitable but infrequent requests that hit global KV but miss SWA KV, it recovers the missing state by recomputing only $n _ { \mathrm { w i n } }$ tokens instead of a full $L \times n _ { \mathrm { w i n } }$ -token forward pass. This bounded replay is the cornerstone of the design: it turns a catastrophic miss into a graceful, inexpensive degradation, thereby justifying the removal of SWA KV from the persistent KV cache.
4. 驱逐 SWA 必有 miss, 靠轻量回退 Encoder SWA Bounded Replay(§3.2.2)压成本: 命中 global 但未命中 SWA 的少数请求, 只重算 $n_{\mathrm{win}}$ 而非 $L\times n_{\mathrm{win}}$. 有界回放是设计基石-- 把灾难性 miss 变成可承受降级, 才敢把 SWA 移出 persistent.

3.2.2. SWA Bounded Replay

3.2.2. SWA Bounded Replay

Since SWA dependencies accumulate across layers, exactly reconstructing the SWA KV of 𝐿 layers would require replaying $L \times n _ { \mathrm { w i n } }$ tokens. SWA Bounded Replay instead replays only the most recent $n _ { \mathrm { w i n } }$ tokens and truncates SWA to the replay segment, accepting approximate states: for a replay starting at position 𝑠, a query at position 𝑖 attends to SWA keys in [max(𝑠, 𝑖 − 𝑊 + 1), 𝑖].

精确重建 $L$ 层 SWA KV 要回放 $L\times n_{\mathrm{win}}$; 有界回放只回放最近 $n_{\mathrm{win}}$, 并把 SWA 截到回放段, 接受近似: 回放起点 $s$ 时, 位置 $i$ 的 query 看 $[\max(s, i-W+1), i]$ 上的 SWA key.

Encoder SWA Bounded Replay. Encoder SWA Bounded Replay makes prefix caching depend only on global KV, allowing SWA KV to be removed from the persistent KV cache.

When the encoder SWA KV is missing, we replay the last $n _ { \mathrm { w i n } }$ tokens of the cached prefix and process them together with the uncached suffix. The replayed tokens regenerate only SWA KV, reusing the cached global KV without recomputation or overwriting, while the uncached suffix generates both global KV and SWA KV.

By design, the replayed prefix state is approximate, so the global KV and SWA KV computed for the uncached suffix depend on the cache-hit position and are not mathematically identical across positions. Encouragingly, our experimental evidence confirms that this bounded replay strategy barely compromises response quality.

Encoder SWA Bounded Replay. 让前缀缓存只依赖 global KV, 从而可从 persistent 去掉 SWA. 缺 encoder SWA 时, 回放缓存前缀末 $n_{\mathrm{win}}$, 与未缓存后缀一起处理: 回放段只重生 SWA, 复用已缓存 global(不重算, 不覆盖); 后缀同时生成 global 与 SWA. 设计上前缀状态近似, 故后缀上算得的 KV 会随命中位置而变, 数学上不完全同一-- 实验显示对回复质量几乎无伤.

Decoder SWA Bounded Replay. Decoder SWA Bounded Replay bounds the decoder forward pass to $n _ { \mathrm { w i n } }$ tokens, nearly halving total prefill computation.

Under CED, decoder global KV is projected from the final encoder hidden states. The only obstacle to ending prefill at the encoder is decoder SWA KV, which is generated from each decoder layer’s own hidden states and is needed by the first decode steps. Since we never cache decoder SWA KV, exactly reconstructing it requires running the $\frac { L } { 2 }$ decoder layers over the last $\textstyle { \frac { L } { 2 } } \times n _ { \operatorname* { w i n } }$ prompt tokens, which is expensive when a short uncached suffix follows a long cached prefix. Therefore, we also apply the bounded replay strategy to this scenario: at every prefill, we replay the last $n _ { \mathrm { w i n } }$ tokens of the prompt, feed their encoder outputs through the decoder layers under the same SWA truncation, and use the resulting decoder SWA KV only for decoding, not for prefix caching.

By design, the reconstructed decoder SWA KV is not mathematically equivalent to that from a full decoder forward pass. Also, we find that this strategy has only a negligible impact on response quality. For added safety, we additionally simulate the same replay during post-training for train-aware adaptation.

Decoder SWA Bounded Replay. 把 decoder 前向限制在 $n_{\mathrm{win}}$ token, 总 prefill 近乎减半. CED 下 decoder global KV 由 encoder 末隐状态投影; 挡住「prefill 停在 encoder」的只剩 decoder SWA(来自各 decoder 层隐状态, 首批 decode 需要). 从不缓存 decoder SWA 时, 精确重建要对末 $\frac{L}{2}\times n_{\mathrm{win}}$ prompt token 跑半网 decoder-- 长前缀命中, 短后缀未缓存时很贵. 故每次 prefill 回放 prompt 末 $n_{\mathrm{win}}$, 把其 encoder 输出在同样 SWA 截断下过 decoder, 所得 decoder SWA 只用于 decode, 不进前缀缓存. 数学上不等价于完整 decoder 前向, 但对质量几乎无伤; 后训练再模拟同款回放, 做 train-aware 适配.

4. Pre-Training 预训练

4.1. Data Construction 数据构建

Text Data Curation In pursuit of higher intelligence, we go beyond the general, sample-level quality reflected by small-scale data experiments and focus more on the holistic interactions among diverse corpora that offer unique information gains. We adopt a more systematic and standardized data construction pipeline to improve data quality and optimize the data mixture. Specifically, based on more comprehensive evaluations, a scaling ladder over model parameters and training data is carefully designed to guide large-scale training runs. We filter out modelgenerated content with limited information gain, including outputs from less capable models and low-quality machine-translated text. We regard such content as implicit duplication, as

文本数据策展 不止小规模实验里的样本级质量, 更盯多样语料间能提供独特信息增益的整体交互. 流水更系统, 标准化; 用更全面评测设计「参数–数据」缩放阶梯, 指导大规模训练. 滤掉信息增益有限的模型生成内容(弱模型输出, 低质机翻), 视之为隐式重复, 因其

it largely reformulates existing information and may become detrimental over long training horizons. We also explore model-in-the-loop data iteration approaches as a foundation for future large-scale synthetic data. In addition, we involve more domain experts to construct fine-grained data quality evaluation dimensions. Compared with the previous version, the new corpus incorporates more recent code from newly released open-source repositories, commits, libraries, and emerging frameworks to cover a broader range of programming languages and better reflects contemporary real-world software engineering scenarios.

大多只是改写已有信息, 长训程上可能有害. 也探索 model-in-the-loop 数据迭代, 为未来大规模合成打底; 并引入更多域专家细化质量评价维度. 相对上一版, 新语料纳入更多新开源仓, commit, 库与新兴框架的近期代码, 覆盖更广语言, 更贴当代软件工程.

Multimodal Data Curation Our multimodal pre-training dataset primarily comprises three types of data: image-text pairs, interleaved image-text data, and domain-specific data. Operating on the premise that raw web data naturally provides rich multimodal knowledge, we refrained from large-scale data synthesis; instead, we prioritized cleaning and utilizing the data in its native form to achieve the most direct and scalable visual knowledge compression during pre-training. During the initial data collection, we found that our crawling system was overly biased toward text-centric web content; we therefore re-bootstrapped it from Common Crawl to improve its coverage of multimodal sources. For image-text data, we extract images together with their associated alt text from webpages, filter them by applying an image-text relevance threshold, and deduplicate them based on image semantics. For interleaved data, we build this subset predominantly from webpages and PDFs. Processing large-scale multimodal corpora usually incurs higher CPU and disk storage costs than processing text-only corpora, which motivated us to organize the interleaved-data construction into progressively more expensive stages. Before image retrieval, we apply heuristic and statistical filtering, deduplication, and quality models to select high-value documents. The surviving documents are then assembled into interleaved image–text sequences, where filtering and deduplication are applied again in an image-aware manner. Finally, we employ SmolVLM (Marafioti et al., 2025) to conduct strict quality scoring on the image-text content, thereby extracting high-quality interleaved data. Documents filtered out during this process are partly recycled into additional image-text pairs via screening and recombination. To compensate for the inherent limitations of web-gathered data, we also incorporate domain-specific datasets to boost the model’s capabilities in finegrained visual perception (e. g., visual grounding and pointing), optical character recognition (OCR), and the acquisition of long-tail knowledge. We also collect extensive image-code pairs and computer-use trajectories to improve multimodal agentic understanding.

多模态数据策展 三类为主: 图文对, 交错图文, 域专用数据. 前提是原始网页已含丰富多模态知识, 故不做大规模合成, 优先清洗并用原生形态做最直接, 可扩展的视觉知识压缩. 初采发现爬虫偏文本站, 于是从 Common Crawl 重引导以覆盖多模态源. 图文对: 抽图与 alt, 相关性阈值过滤, 按图像语义去重. 交错数据主要来自网页与 PDF; 因多模态处理 CPU/磁盘更贵, 交错构建分递进昂贵阶段: 取图前先启发式/统计过滤, 去重与质量模型筛高价值文档; 幸存文档再组装成交错序列, 并做图像感知过滤去重; 最终用 SmolVLM 严打分抽高质量交错. 筛掉文档部分经筛选重组回收为额外图文对. 为补网页局限, 再加域数据抬细粒度视觉感知(定位/指点), OCR 与长尾知识; 并大量收集图–代码对与 computer-use 轨迹, 加强多模态 Agent 理解.

Data Integration and Deduplication As our text-only and multimodal data were processed through distinct pipelines, we constructed the final training corpus as the union of both data sources. For overlapping samples, we replace the text-only versions with their multimodal counterparts and use the larger epoch count of the two configurations. After this substitution, the resulting corpus uses a 7: 1 token ratio of text-only to multimodal data. We minimize sample overlap during pre-training and context extension by jointly prefetching and assigning training samples. Ultra-long documents are deterministically pre-split before mixing to ensure a uniform distribution of training tokens across data shards and training steps. We further enhance our best-fit packing algorithm, achieving a padding rate of at most 10⁻⁴.

数据整合与去重 文本与多模态分流水, 最终语料取并集; 重叠样本用多模态版替换纯文本版, epoch 取两者较大值. 替换后文本: 多模态 token 比 7: 1. 预训练与上下文扩展期联合预取分配样本, 压重叠; 超长文档混洗前确定性预切, 保证各 shard/步 token 均匀. best-fit packing 再加强, padding 率至多 $10^{-4}$.

4.2. Pre-Training Setups 预训练设定

4.2.1. Model Setups 模型设定

We set the number of Transformer layers to 40 and the hidden dimension 𝑑 to 5120. We adopt a Causal Encoder-Decoder architecture, with 20 layers in the encoder and 20 layers in the

Transformer 40 层, 隐维 $d=5120$. CED: encoder / decoder 各 20 层.

decoder. For the first two layers, we use pure sliding window attention. The remaining 18 encoder layers use CSA2 with a compression rate of $m = 2$ . These layers are divided into three identically configured groups of six layers. In each group, the first layer operates in Full Mode, and the remaining five layers operate in Reuse Mode. The 20 decoder layers use CSA2 with a compression rate of $m = 1$ . These layers are divided into five groups of four layers. In the first group, the first layer operates in Full Mode, and the remaining three layers operate in Reuse Mode. The remaining four groups share the same configuration: the first layer operates in Reindex Mode, and the remaining three layers operate in Reuse Mode. For all CSA2 layers, we set the number of indexer query heads to 32, the indexer head dimension to 128, and the number of KV entries selected for sparse attention (i. e., attention top-k) to 512. We set the number of query heads to 64, the head dimension to 512, and the query compression dimension to 1280. For the Hierarchical Sparse Indexer, we select a maximum of 2, 048 blocks with 8 positions, yielding up to 16, 384 candidate positions in total. The number of output projection groups is set to 8, and the dimension of each intermediate attention output is set to 1024. For the additional branch of sliding window attention, the window size $n _ { \mathrm { w i n } }$ is set to 128. We employ MoE layers in all Transformer blocks, using SwiGLU activation function with clamping (OpenAI, 2025) at a threshold of 10. Each MoE layer consists of 1 shared expert and 384 routed experts, where the intermediate hidden dimension of each expert is 2304. Among the routed experts, 6 experts will be activated for each token. As for mHC, the expansion factor is set to 4, and the number of Sinkhorn-Knopp iterations is set to 20. For the vision encoder, we set its number of layers to 32, the hidden dimension to 1024, the number of attention heads to $1 6 ,$ and the image patch size to 14. The vision MLP projector has 2 layers with a hidden dimension of 5120. Under this configuration, DeepSeek-V4.1-Flash comprises 552B backbone parameters, with 8B activated per token during prefill and 16B during decode.

前两层纯 SWA; 其余 18 个 encoder 层 CSA2, $m=2$, 分三组各 6 层: 每组首层 Full, 后五层 Reuse. 20 个 decoder 层 CSA2, $m=1$, 五组各 4 层: 第一组首 Full + 三 Reuse; 后四组首 Reindex + 三 Reuse. CSA2: indexer query 头 32, 头维 128, 稀疏注意力 top-k=512; query 头 64, 头维 512, query 压缩维 1280. 分层稀疏 Indexer: 最多 2048 块×8 位置 → 至多 16384 候选. 输出投影组 8, 中间注意力输出维 1024. SWA 窗 $n_{\mathrm{win}}=128$. 全层 MoE: SwiGLU 钳位阈值 10; 1 共享 + 384 路由, 专家中间维 2304, 每 token 激活 6 个路由专家. mHC 扩展因子 4, Sinkhorn-Knopp 迭代 20. 视觉编码器 32 层, 隐维 1024, 头 16, patch 14; MLP 投影 2 层, 隐维 5120. 此配置下骨干 552B, prefill 激活 8B, decode 激活 16B.

4.2.2. Training Setups 训练设定

We employ the Muon optimizer (Jordan et al., 2024; Liu et al., 2025) for the parameters of linear transformations, use AdamW optimizer (Loshchilov and Hutter, 2019) for the weights of all RMSNorm modules and other non-matrix parameters, and use Sinkhorn-balanced update for all embeddings and prediction head. For AdamW, we set its hyper-parameters to $\beta _ { 1 } = 0 . 9 , \beta _ { 2 } = 0 . 9 5 ,$ $\varepsilon = 1 0 ^ { - 2 0 }$ , and weight_decay = 0.1. For Muon, we set the momentum to 0.95 and the weight decay to 0.1, and rescale the RMS of each update matrix to 0.18 for reutilization of the AdamW learning rate. For Sinkhorn-balanced updating, we use the same momentum coefficient and learning-rate correction factor as for Muon and set $K = 1 1 , \tau = 1 0 ^ { - 3 } , \varepsilon = 1 0 ^ { - 2 0 }$ . Following (Cheng et al., 2026b), the learning rate of Engram is scaled by 5×. We train DeepSeek-V4.1-Flash on 45T tokens of multimodal data with no instability. We keep the batch size fixed at 100.6 million tokens throughout training. The learning rate is linearly warmed up over the first 2000 steps and then maintained at $\tilde { 2 . 6 \times 1 0 ^ { - 4 } }$