arXiv:2602.21548v2 [cs.DC] 26 Feb 2026

DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference · DualPath: 打破 Agent 推理中的存储带宽瓶颈

Yongtong Wu^1,3 Shaoyuan Chen^2,3 Yinmin Zhong^1,3 Rilin Huang¹ Yixuan Tan³ Wentao Zhang³ Liyue Zhang³ Shangyan Zhou³ Yuxuan Liu³ Shunfeng Zhou³ Mingxing Zhang² Xin Jin1 Panpan Huang

¹School of Computer Science, Peking University ²Tsinghua University ³DeepSeek-AI

ABSTRACT

The performance of multi-turn, agentic LLM inference is increasingly dominated by KV-Cache storage I/O rather than computation. In prevalent disaggregated architectures, loading the massive KV-Cache from external storage creates a fundamental imbalance: storage NICs on prefill engines be come bandwidth-saturated, while those on decoding engines remain idle. This asymmetry severely constrains overall system throughput.

多轮的 Agent 式 LLM 推理, 性能越来越取决于 KV-Cache 的存储 I/O, 而不是计算. 在目前主流的分离式架构里, 从外部存储加载海量 KV-Cache 会造成一种根本性的失衡: prefill 引擎上的存储网卡带宽被打满, decode 引擎上的存储网卡却闲着. 这种不对称严重限制了整个系统的吞吐.

We present DualPath, an inference system that breaks this bottleneck by introducing dual-path KV-Cache loading. Beyond the traditional storage-to-prefill path, DualPath enables a novel storage-to-decode path, in which the KV-Cache is loaded into decoding engines and then efficiently transferred to prefill engines via RDMA over the compute network. DualPath combines this optimized data path — which inherently avoids network congestion and avoids interference with latency-critical model execution communications — with a global scheduler that dynamically balances load across prefill and decode engines.

我们提出 DualPath, 一个用双路径 KV-Cache 加载打破这一瓶颈的推理系统. 除了传统的「存储到 prefill」路径, DualPath 新开一条「存储到 decode」路径: KV-Cache 先加载进 decode 引擎, 再经计算网络用 RDMA 高效地传给 prefill 引擎. DualPath 把这条优化过的数据通路 (它本身不会造成网络拥塞, 也不干扰对延迟敏感的模型执行通信) 与一个全局调度器结合起来, 由调度器在 prefill 与 decode 引擎之间动态均衡负载.

Our evaluation on three models with realistic agentic workloads demonstrates that DualPath improves offline inference throughput by up to 1.87× on our in-house inference system. It can also improve online serving throughput by an average factor of 1.96× without violating SLO.

我们在三个模型和真实 Agent 负载上做了评测: 在我们的内部推理系统上, DualPath 把离线推理吞吐最多提高到 1.87 倍; 在线服务吞吐在不违反 SLO 的前提下平均提高到 1.96 倍.

1 INTRODUCTION

Large Language Models (LLMs) are rapidly evolving from single-turn chatbots [16, 36] and standalone reasoners [36] into agentic systems that can autonomously plan, invoke tools, and solve real-world tasks through multi-turn interactions [8, 24, 31, 45, 48]. In such settings, an LLM no longer serves isolated prompts; instead, it participates in long-running sessions where context accumulates over time [29]. As agentic applications become increasingly prevalent, multi-turn LLM inference has emerged as a critical workload in production systems, ranging from coding assistants [47, 52] to autonomous task agents [28, 59].

大语言模型 (LLM) 正从单轮聊天机器人 [16, 36] 和独立的推理器 [36] 快速演变为 Agent 系统: 能自主规划, 调用工具, 通过多轮交互解决真实任务 [8, 24, 31, 45, 48]. 在这种场景里, LLM 不再处理彼此孤立的 prompt, 而是参与长时间运行的会话, 上下文随时间不断累积 [29]. 随着 Agent 应用越来越普遍, 多轮 LLM 推理已成为生产系统里的关键负载, 从编程助手 [47, 52] 到自主任务 Agent [28, 59] 都是如此.

Figure 1: Existing bottleneck (left) and DualPath (right).

This paradigm shift in applications has driven a significant transformation in LLM inference workloads: from traditional human-LLM interaction to human-LLM-environment interaction, called the agentic paradigm. The typical pattern of human-model interaction involves users providing input, engaging in a few rounds of interaction with the LLM, and consuming the results generated by the LLM. By contrast, an agentic LLM may interact with an external environment, through tools such as a web browser and Python interpreter, over dozens or even hundreds of turns. Although each individual tool call or feedback is short (often hundreds of tokens), the context accumulates across turns and can grow to extreme lengths. As a result, agentic workloads become highly I/O-bound: the multi-turn, short-append pattern leads to very high KV-Cache hit rates — typically ≥ 95% [7] — making the efficiency of KV-Cache loading, rather than pure computation, the dominant performance factor.

应用范式的这一转变, 让 LLM 推理负载发生了明显变化: 从传统的「人与 LLM」交互, 变成「人, LLM 与环境」的交互, 称为 Agent 范式. 典型的人机交互是: 用户给出输入, 与 LLM 来回几轮, 然后使用 LLM 生成的结果. 相比之下, Agent 式 LLM 会借助网页浏览器, Python 解释器这类工具与外部环境交互, 持续几十甚至上百轮. 每一次工具调用或反馈都很短 (常常只有几百个 token), 但上下文跨轮累积, 可以长到极端的程度. 于是 Agent 负载变得高度受 I/O 限制: 多轮, 短追加的模式带来极高的 KV-Cache 命中率 (通常 ≥ 95% [7]), 决定性能的主要因素变成 KV-Cache 的加载效率, 而不是单纯的计算.

To improve throughput under agentic workloads, existing LLM inference systems have converged on a common set of architectural patterns: layer-wise prefill [17, 50], prefill– decode (PD) disaggregation [37, 55, 58], and external KV-Cache storage [18, 30, 38]. In these systems, prefill engines load the KV-Cache in a layer-wise manner to accommodate as many requests as possible within a single batch. When prefill completes, decoding engines typically receive KV-Cache from prefill engines via a high-performance RDMA network. The decoding engines then generate tokens and store their KV-Cache in distributed storage to enable reuse across turns.

为提高 Agent 负载下的吞吐, 现有 LLM 推理系统收敛到一组共同的架构模式: 逐层 prefill [17, 50], prefill-decode (PD) 分离 [37, 55, 58], 以及外部 KV-Cache 存储 [18, 30, 38]. 在这些系统里, prefill 引擎逐层加载 KV-Cache, 以便在一个 batch 里容纳尽可能多的请求. prefill 结束后, decode 引擎通常经高性能 RDMA 网络从 prefill 引擎接收 KV-Cache, 随后生成 token, 并把自己的 KV-Cache 存进分布式存储, 供后续轮次复用.

However, this architecture also introduces a critical limitation. As shown in Figure 1, prefill engines must load large

volumes of KV-Cache from remote storage. As a result, prefillside storage network bandwidth becomes the throughput bot tleneck of the entire system, even though decoding engines often have substantial unused storage network bandwidth.

但这种架构也带来一个关键局限. 如图 1 所示, prefill 引擎必须从远端存储加载大量 KV-Cache. 结果 prefill 侧的存储网络带宽成了整个系统的吞吐瓶颈, 尽管 decode 引擎往往还有大量没用上的存储网络带宽.

This imbalance reveals a fundamental inefficiency in existing designs: storage network bandwidth is unevenly utilized across engines. The bandwidth of prefill engines are persistently saturated, while decoding engines remain underutilized. Simply provisioning more bandwidth to prefill engines is costly and often impractical in general-purpose clusters. Therefore, it is promising to exploit and combine the available I/O bandwidth of all engines, rather than overloading prefill engines alone, to accelerate KV-Cache loading for agentic LLM workloads.

这种失衡暴露了现有设计的一个根本低效: 各引擎对存储网络带宽的利用很不均匀. prefill 引擎的带宽长期打满, decode 引擎却用得很少. 单纯给 prefill 引擎配更多带宽成本高, 在通用集群里也常常不现实. 因此, 更有希望的做法是把所有引擎可用的 I/O 带宽都利用起来并汇总使用, 而不是只压在 prefill 引擎上, 以此加速 Agent 负载的 KV-Cache 加载.

Prior studies have attempted to alleviate the KV-Cache loading bottleneck. Mooncake [38] caches KV-Cache in a distributed DRAM pool and employs an affinity-aware scheduler to maximize the DRAM KV-Cache hit rate. However, it cannot be used in memory-constrained scenarios, such as the rollout phase in RL, where DRAM is occupied to hold large training state that is offloaded from HBM. It is also not cost-effective in scenarios with enormous working sets (e.g., online serving), considering the cost comparison between DRAM and SSD. Other attempts reduce the amount of KV-Cache data to retrieve [19] and reduce the retrieval overhead [22, 51]. However, they do not solve the inherent inefficiency caused by storage I/O imbalance between different engines.

已有研究尝试缓解 KV-Cache 加载瓶颈. Mooncake [38] 把 KV-Cache 缓存在分布式 DRAM 池里, 并用亲和性感知的调度器尽量提高 DRAM 中 KV-Cache 的命中率. 但它用不了内存受限的场景, 例如 RL 的 rollout 阶段: 那时 DRAM 被从 HBM 卸载下来的大量训练状态占着. 在工作集极大的场景 (例如在线服务) 里, 考虑到 DRAM 与 SSD 的成本差, 它也不划算. 另一些工作减少需要取回的 KV-Cache 数据量 [19], 或降低取回开销 [22, 51]. 但这些都没有解决不同引擎之间存储 I/O 失衡造成的内在低效.

In this paper, we present DualPath, a new LLM inference system that rethinks KV-Cache loading in modern inference architectures for agentic workloads. The key insight behind DualPath is that KV-Cache loading does not have to be prefillcentric. While existing systems always load KV-Cache directly from storage into prefill engines, they cannot utilize the remote storage bandwidth of decoding engines. Dual-Path leverages this observation by enabling dual-path KV-Cache loading: in addition to the conventional storage-to-prefill path, KV-Cache can be loaded into decoding engines and then transferred to prefill engines via high-performance RDMA. By dynamically selecting between these paths, DualPath redistributes network load and alleviates prefill-side bandwidth pressure.

本文提出 DualPath, 一个新的 LLM 推理系统, 针对 Agent 负载重新设计现代推理架构里的 KV-Cache 加载. DualPath 背后的关键观察是: KV-Cache 加载不必以 prefill 为中心. 现有系统总是把 KV-Cache 从存储直接加载进 prefill 引擎, 于是用不上 decode 引擎的远端存储带宽. DualPath 利用这一点, 实现双路径 KV-Cache 加载: 除了传统的「存储到 prefill」路径, KV-Cache 还可以先加载进 decode 引擎, 再经高性能 RDMA 传给 prefill 引擎. 通过在两条路径之间动态选择, DualPath 重新分配网络负载, 减轻 prefill 侧的带宽压力.

Realizing this design raises two challenges. First, introducing an extra loading path introduces complex traffic patterns and potential interference with collective primitives in model execution, which can degrade overall performance if unman aged. Second, the system must decide online which loading path to use under dynamic and heterogeneous workloads, and ensure load balance across both GPUs and NICs simultaneously. To address these challenges, DualPath adopts (1) an optimized dual-path loading data path design, which introduces no inherent congestion under common P/D ratios,

(2) a NIC-centric traffic management approach to isolate KV-Cache traffic from latency-sensitive model inference communications, and (3) a dynamic scheduling policy that jointly balances computation and network utilization across prefill and decoding engines.

实现这一设计有两个挑战. 第一, 多一条加载路径会带来复杂的流量模式, 并可能干扰模型执行中的集合通信原语, 不加管理会拖累整体性能. 第二, 系统必须在动态, 异构的负载下在线决定用哪条加载路径, 并同时保证 GPU 和网卡两方面的负载均衡. 为此 DualPath 采用: (1) 优化过的双路径加载数据通路设计, 在常见的 P/D 比例下本身不会引入拥塞; (2) 以网卡为中心的流量管理, 把 KV-Cache 流量与延迟敏感的模型推理通信隔离开; (3) 动态调度策略, 在 prefill 与 decode 引擎之间同时均衡计算与网络利用率.

We implement DualPath on top of a modern inference stack and evaluate it using representative agentic workloads with long contexts and high cache reuse. Experiments show that DualPath significantly improves system throughput and the first token latency, while maintaining the latency between tokens. In agentic inference scenarios, DualPath increases end-to-end throughput by up to 1.87× for offline inference, and improves the online serving throughput by 1.96× on average.

我们在一套现代推理栈之上实现了 DualPath, 并用长上下文, 高缓存复用的代表性 Agent 负载评测. 实验表明, DualPath 明显提高了系统吞吐并降低了首 token 延迟, 同时保持 token 间延迟不变. 在 Agent 推理场景下, DualPath 把离线推理的端到端吞吐最多提高到 1.87 倍, 把在线服务吞吐平均提高到 1.96 倍.

In summary, this paper makes three contributions:

总结起来, 本文有三项贡献:

• We identify the I/O-bound nature of multi-turn, agentic LLM workloads and show that KV-Cache loading dominates system performance under modern LLM inference architectures.

• 我们指出多轮 Agent 式 LLM 负载受 I/O 限制的性质, 并说明在现代 LLM 推理架构下, KV-Cache 加载主导系统性能.

• We present DualPath, an inference system that introduces dual-path KV-Cache loading and leverages decoding-engine bandwidth to resolve prefill-side bottlenecks.

• 我们提出 DualPath, 一个引入双路径 KV-Cache 加载, 借用 decode 引擎带宽来解决 prefill 侧瓶颈的推理系统.

• We design and evaluate a workload-aware scheduling algorithm that dynamically balances computation and network resources, significantly improving balance on realistic workloads.

• 我们设计并评测了一种感知负载的调度算法, 动态均衡计算与网络资源, 在真实负载上明显改善了均衡程度.

2 BACKGROUND · 背景

2.1 LLM Inference Preliminary · LLM 推理基础

LLM inference is becoming one of the most important system workloads recently. Popular LLMs utilize decoder-only transformer architecture, comprising stacked blocks with attention layers and feed-forward networks (FFNs). Attention layers enable token interactions within requests, while FFNs process tokens independently. The model predicts subsequent tokens based on preceding ones, storing attention keys and values as KV-Cache in HBM to avoid recomputations.

LLM 推理近来正成为最重要的系统负载之一. 主流 LLM 采用 decoder-only Transformer 架构, 由堆叠的块组成, 每块含注意力层和前馈网络 (FFN). 注意力层让同一请求内的 token 相互作用, FFN 则独立处理每个 token. 模型根据前面的 token 预测后续 token, 并把注意力的 key 和 value 作为 KV-Cache 存在 HBM 里, 避免重复计算.

PD-disaggregated Inference. Prefill–decode (PD) disaggregation [37, 58] separates the prefill phase from the decode phase, assigning them to dedicated prefill engines (PEs) and decode engines (DEs), respectively. The two phases exhibit distinct compute and memory patterns: prefill is computeintensive and batched, while decode is memory-bound and latency-sensitive. With PD disaggregation, PEs load the hit KV-Cache and perform prefilling; then, they transfer the KV-Cache to DEs, which perform autoregressive decoding. This design reduces interference between phases, enables stagespecific optimizations, and improves scalability, making it

Figure 2: Agent trajectory example.

the de facto architecture for modern LLM serving. To support multi-turn conversations, the KV-Cache is often stored in distributed storage for reuse across turns.

PD 分离推理. prefill-decode (PD) 分离 [37, 58] 把 prefill 阶段和 decode 阶段拆开, 分别交给专用的 prefill 引擎 (PE) 和 decode 引擎 (DE). 两个阶段的计算与访存特征不同: prefill 是计算密集, 成批处理的; decode 受访存带宽限制, 对延迟敏感. 在 PD 分离下, PE 加载命中的 KV-Cache 并做 prefill, 然后把 KV-Cache 传给 DE, 由 DE 做自回归解码. 这种设计减少了阶段间干扰, 允许针对各阶段做专门优化, 也提高了可扩展性, 因此成为现代 LLM 服务事实上的标准架构. 为支持多轮对话, KV-Cache 常存放在分布式存储里, 供跨轮复用.

Layerwise Prefill. Long-context prefill is bottlenecked by HBM capacity, as both activations and the KV-Cache for the entire batch must reside within it, forcing limited batch sizes and leading to poor GPU utilization. LayerKV [50] and PrefillOnly [17] address this problem by exploiting the strong locality in prefill computations: each layer requires only its own layer-specific KV-Cache. Consequently, the KV-Cache can be allocated and freed per layer, and the GPU holds only one layer’s KV-Cache for the forward batch. This increases the effective batch size (in tokens) by approximately a factor equal to the number of layers, boosting prefill throughput.

逐层 prefill. 长上下文 prefill 受 HBM 容量限制: 整个 batch 的激活和 KV-Cache 都必须放在 HBM 里, batch 大小因此受限, GPU 利用率低. LayerKV [50] 和 PrefillOnly [17] 利用 prefill 计算的强局部性解决这个问题: 每一层只需要本层的 KV-Cache. 于是 KV-Cache 可以按层分配, 按层释放, GPU 上只保留当前前向 batch 一层的 KV-Cache. 这使有效 batch 大小 (按 token 计) 大约扩大到层数那么多倍, 提高了 prefill 吞吐.

2.2 Agentic Use of LLMs · LLM 的 Agent 式使用

LLMs increasingly power agentic applications that perform multi-turn reasoning and interact with an environment (via e.g., terminal commands, code execution, or asking for human feedback) over long sessions. As shown in Figure 2, in a typical turn, the model receives a prompt composed by the previous context plus some newly appended tokens (often tool output or user input) and generates the next action or response. A single agent run is a trajectory of dozens or even hundreds of turns: the context grows turn-by-turn and can reach up to one million tokens [4, 11]. Because most of the context, typically >95% tokens in our traces, is reused across rounds, the vast majority of tokens in each round can hit the KV-Cache; only the newly appended context needs prefill computation. Due to the extreme length of agent trajectories, DRAM and HBM-based KV-Cache storage like Mooncake [38] can only store a small proportion of KV-Caches, necessitating the use of larger yet cheaper external SSD-based KV-Cache storage [13].

LLM 越来越多地驱动 Agent 应用: 在长会话中做多轮推理, 并与环境交互 (例如执行终端命令, 运行代码, 请求人类反馈). 如图 2 所示, 典型的一轮里, 模型收到的 prompt 由之前的上下文加上一些新追加的 token (通常是工具输出或用户输入) 组成, 模型据此生成下一个动作或回复. 一次 Agent 运行是一条几十甚至上百轮的轨迹: 上下文逐轮增长, 最长可到一百万 token [4, 11]. 由于大部分上下文 (在我们的 trace 里通常超过 95% 的 token) 跨轮复用, 每轮绝大多数 token 都能命中 KV-Cache, 只有新追加的部分需要 prefill 计算. 由于 Agent 轨迹极长, Mooncake [38] 这类基于 DRAM 和 HBM 的 KV-Cache 存储只能放下很小一部分 KV-Cache, 因此需要容量更大, 价格更低的外部 SSD 型 KV-Cache 存储 [13].

The agentic LLM inference workload is also prevalent in agent LLM training, which often adopts reinforcement learning (RL) approaches. In a typical RL training loop, the agent LLM first undergoes a rollout phase, where it is prompted to generate a large number of multi-step agent trajectories.

These trajectories are then scored by a separate reward model. Finally, the LLM parameters are updated to increase the likelihood of high-scoring outputs and reduce the likelihood of low-scoring ones. During the rollout phase, substantial data (like reward model and optimizer states) is offloaded to host DRAM, further constraining the available DRAM for KV-Cache. This reinforces the need for external, high-capacity KV-Cache storage that can accommodate long agentic rollout contexts efficiently.

Agent 式 LLM 推理负载在 Agent LLM 的训练里也很常见, 这类训练常用强化学习 (RL). 典型的 RL 训练循环里, Agent LLM 先经历 rollout 阶段, 按提示生成大量多步 Agent 轨迹; 随后由单独的奖励模型给轨迹打分; 最终更新 LLM 参数, 提高高分输出的似然, 降低低分输出的似然. rollout 阶段有大量数据 (如奖励模型和优化器状态) 被卸载到主机 DRAM, 留给 KV-Cache 的 DRAM 更少. 这进一步说明需要外部的大容量 KV-Cache 存储, 能高效容纳长 Agent rollout 的上下文.

2.3 Modern AI Data Center Architecture · 现代 AI 数据中心架构

Modern AI data centers are purpose-built logical supercomputers engineered to handle large-scale generative AI training and inference workloads. For example, in a standard NVIDIA DGX SuperPOD [32], each node is equipped with 8 Hopper GPUs interconnected via high-speed NVLink. Each GPU is paired with a dedicated 400 Gbps compute NIC (CNIC, also known as east-west NIC), which maximizes inter-node communication bandwidth. Independent of the compute fabric, each node also features a storage NIC (SNIC, also known as south-north NIC) up to 400 Gbps, providing fast access to datasets, model checkpoints, and on-disk KV cache.

现代 AI 数据中心是专门建造的逻辑超级计算机, 用来承担大规模生成式 AI 的训练与推理负载. 以标准的 NVIDIA DGX SuperPOD [32] 为例, 每个节点配 8 块 Hopper GPU, 由高速 NVLink 互连. 每块 GPU 配一块专用的 400 Gbps 计算网卡 (CNIC, 也叫东西向网卡), 以最大化节点间通信带宽. 与计算网络相互独立, 每个节点还有一块最高 400 Gbps 的存储网卡 (SNIC, 也叫南北向网卡), 用于快速访问数据集, 模型 checkpoint 和落盘的 KV cache.

A fundamental principle of this architecture is that the compute network and the storage network are isolated from each other [55]. This separation is essential to maximize both storage and application performance. By isolating highintensity east-west compute traffic between GPUs from storage traffic, the architecture prevents interference between them, and drastically reduces compute communication latency. This design also ensures that the inter-GPU communication remains highly reliable and predictable even when performing data-intensive tasks such as reading large datasets or writing multi-terabyte model checkpoints.

这种架构的一条基本原则是计算网络和存储网络相互隔离 [55]. GPU 之间高强度的东西向计算流量与存储流量分开后, 两者不再争用同一网络, 计算通信延迟随之降低. 即使系统正在读取大型数据集或写入数 TB 模型 checkpoint, GPU 间通信也能保持较稳定、可预测的表现.

3 BOTTLENECK & MOTIVATION · 瓶颈与动机

We observe severe GPU underutilization during agentic inference tasks. Our investigation reveals that KV-Cache loading speed is the bottleneck due to the limited bandwidth of the single storage NIC on each node. Analysis demonstrates that three decisive factors jointly cause this bottleneck, as discussed below.

我们观察到 Agent 推理任务中 GPU 利用率严重不足. 调查发现, 由于每个节点只有一块存储网卡, 带宽有限, KV-Cache 的加载速度成了瓶颈. 分析表明, 这一瓶颈由三个决定性因素共同造成, 下面逐一讨论.

First, agentic workloads exhibit high KV-Cache hit rates, which require more I/O and less computation, thus creating a severe I/O bottleneck. Agentic workloads are naturally long-context, short-append, and multi-turn. On each turn, the GPU needs to read the KV-Cache of the entire context from persistent storage and perform prefill computation for appended tokens. Our trace collected from representative coding tasks shows the mean number of rounds is 157, demonstrating the tendency of LLMs to engage in multi-turn interactions. The average context length is 32.7k, while the

append length mean is only 429, which means a KV-Cache hit rate of 98.7%. In such a scenario, the cache-compute ratio, defined as the ratio of KV-Cache to load and the computation needed, is approximately 22 GB/PFLOP for DeepSeek-V3.2 [16], posing a significant bottleneck on storage bandwidth. Note that the KV-Cache size of DeepSeek MLA model is already highly optimized; for models with larger KV-Cache sizes (see Table 1), the situation is even worse. The ratio of DeepSeek-V3.2 is higher than DeepSeek-V3 [15], benefiting from its sparse attention design, lowering computation demands.

第一, Agent 负载的 KV-Cache 命中率高, 需要更多 I/O, 更少计算, 因而形成严重的 I/O 瓶颈. Agent 负载天然是长上下文, 短追加, 多轮的. 每一轮, GPU 都要从持久存储读回整段上下文的 KV-Cache, 只对追加的 token 做 prefill 计算. 我们从代表性编程任务里采集的 trace 显示, 平均轮数是 157, 说明 LLM 倾向于多轮交互. 平均上下文长度是 32.7k, 而平均追加长度只有 429, 即 KV-Cache 命中率 98.7%. 在这种场景下, cache-compute 比 (定义为需要加载的 KV-Cache 量与所需计算量之比) 对 DeepSeek-V3.2 [16] 约为 22 GB/PFLOP, 给存储带宽带来很大压力. 要注意 DeepSeek MLA 模型的 KV-Cache 体积已经高度优化过; 对 KV-Cache 更大的模型 (见表 1), 情况更糟. DeepSeek-V3.2 的这一比值高于 DeepSeek-V3 [15], 原因是稀疏注意力设计降低了计算需求.

看表: 表 1 和正文的「约 22 GB/PFLOP」能不能从模型配置复算出来, 分子分母各算了什么? 答: 文中没有给出口径, 以下是按公开配置推出的说法. 分子取整段上下文的 KV 字节, 分母取追加 token 的前向 FLOPs. DeepSeek-V3.2 按 FlashMLA 的 FP8 布局, 每 token 每层主 KV 656 字节, 61 层约 40KB; 每个追加 token 的计算取线性层 $2\times 37\text{B}$, DSA 主注意力每个 query 只看 2048 个 token, 每层约 $2\times2048\times(576+512)\times128$ FLOPs, 再加 indexer 对全部上下文的打分 $2\times64\times128\times L_{ctx}$ 每层. 取 $L_{ctx}=32.7\text{k}$, 追加 429 token, 得 1.31GB 对 0.061 PFLOP, 约 21.6 GB/PFLOP; 同样算法在 16K 与 64K 得 12.2 与 35.1, 与表 1 的 13-36 对得上. Qwen2.5-32B 按 GQA 8 个 KV 头, 128 维, 64 层, FP16 算出 115.8 到 265.4, 与表 1 的 117-267 也吻合. 吻合的前提是追加长度固定为 429 且不计 indexer 的 key cache; 把 indexer 每 token 每层 132 字节算进去, 32.7k 处会变成约 26. 这也说明为什么 V3.2 的比值随上下文增长: 主注意力计算被 top-k 封顶, 要读的 KV 却随长度线性增长.

Second, the hardware evolution trend is not well suited for agentic inference workloads. In recent years, network bandwidth and HBM capacity have lagged behind the growth of GPU FLOPS, which drives us to run into memory and communication walls under agentic workloads. As shown in Figure 3, from NVIDIA Ampere to Blackwell, the I/Ocompute ratio decreases by 14.4×. Low NIC bandwidth limits KV-Cache loading speed, making GPUs idle. In addition, small HBM capacity limits the token batch size for GPU kernels [10, 14, 27, 54] to compute at the same time, hindering full utilization of compute units such as Tensor Core [17].

第二, 硬件演进趋势与 Agent 推理负载不太匹配. 近年来网络带宽和 HBM 容量的增长落后于 GPU FLOPS 的增长, 使我们在 Agent 负载下撞上内存墙和通信墙. 如图 3 所示, 从 NVIDIA Ampere 到 Blackwell, I/O 与计算之比下降了 14.4 倍. 网卡带宽低限制了 KV-Cache 加载速度, GPU 只能空等. 另外, HBM 容量小限制了 GPU kernel [10, 14, 27, 54] 同时计算的 token batch 大小, Tensor Core 这类计算单元难以被充分利用 [17].

Third, existing LLM inference systems exhibit severe storage network utilization imbalance across different engine types. In prevalent PD-disaggregated systems, the KV-Cache for hit tokens is loaded exclusively by prefill engines directly from remote storage. This design centralizes all storage I/O pressure on the prefill-side SNICs, while the SNICs on decode engines remain largely idle. Consequently, the aggregate storage network bandwidth cannot be fully harnessed.

第三, 现有 LLM 推理系统在不同类型引擎之间的存储网络利用严重失衡. 在主流的 PD 分离系统里, 命中 token 的 KV-Cache 只由 prefill 引擎直接从远端存储加载. 这种设计把全部存储 I/O 压力集中到 prefill 侧的 SNIC 上, decode 引擎的 SNIC 大部分时间闲置. 结果存储网络的总带宽无法被充分利用.

The above analysis demonstrates that the fundamental performance issue for agentic inference on PD-disaggregated architecture is the high I/O demand for KV-Cache retrieval and unbalanced storage network bandwidth utilization across inference engines. Meanwhile, we observe that the network traffic of the compute network, which has much larger aggregate bandwidth than the storage network, exhibits an intermittent pattern: collective operations used in model inference burst in sub-millisecond intervals. Therefore, an

Figure 3: Left: Hardware trends of NVIDIA GPUs. Right: Relative token throughput with varying request batch size (each request has 30K context with 300 tokens appended).

opportunity naturally emerges: we can utilize the SNIC bandwidth of decode nodes to load KV-Cache from storage, and transfer it back to the prefill nodes, utilizing the spare bandwidth of the faster compute network.

以上分析说明, PD 分离架构上 Agent 推理的根本性能问题在于: KV-Cache 读取的 I/O 需求高, 而各推理引擎对存储网络带宽的利用不均衡. 与此同时, 我们观察到计算网络 (总带宽远大于存储网络) 的流量呈间歇模式: 模型推理用到的集合通信在亚毫秒级的时间间隔内突发. 于是一个机会自然出现: 可以用 decode 节点的 SNIC 带宽从存储加载 KV-Cache, 再借更快的计算网络的空闲带宽把它传回 prefill 节点.

核对: 图 3 左的「GPU Compute: 28.8x」是同一精度下的比较吗? 14.4 倍从哪来? 答: 14.4 就是 $28.8/2.0$, 即 FLOPS 增长倍数除以网卡 (图中同时标 PCIe) 带宽增长倍数. 28.8 这个数对得上 $9.0/0.3125$: 0.3125 PFLOPS 是 A100 的稠密 BF16 峰值, 9 PFLOPS 是 B200 的稠密 FP4 峰值, 2022 年那一点约 2 PFLOPS 对应 H100 的稠密 FP8. 图中三个点各取当代最低精度, 所以 28.8 倍里有一部分来自精度从 16 位降到 4 位, 同精度比较的增长倍数会小很多. 文中没有写明精度口径. 结论方向不变, 但 14.4 倍是偏大的上限.

4 DUALPATH SYSTEM OVERVIEW · DualPath 系统概览

To break the prefill-side storage I/O bottleneck, we propose a dual-path loading architecture that fundamentally rethinks how KV-Cache is retrieved in PD-disaggregated inference. Based on this architecture, we design and implement DualPath. DualPath adopts two widely-adopted techniques demonstrated in §2: (1) PD Disaggregation [37, 58], which separates prompt and decode processing for better efficiency. (2) Layerwise prefill, which avoids HBM bottlenecks recognized by LayerKV [50] and PrefillOnly [17] on prefill engines and improves GPU utilization.

为打破 prefill 侧的存储 I/O 瓶颈, 我们提出双路径加载架构, 从根本上重新设计 PD 分离推理中 KV-Cache 的读取方式. 基于这一架构, 我们设计并实现了 DualPath. DualPath 采用 §2 介绍过的两项广泛使用的技术: (1) PD 分离 [37, 58], 把 prompt 处理与 decode 拆开以提高效率; (2) 逐层 prefill, 避免 LayerKV [50] 和 PrefillOnly [17] 指出的 prefill 引擎 HBM 瓶颈, 提高 GPU 利用率.

Our system consists of the following components:

系统由以下组件构成:

• Inference Engines. Each engine manages one GPU. Engines are categorized into prefill engines (PEs) for prefill and decoding engines (DEs) for decode.

• 推理引擎. 每个引擎管理一块 GPU. 引擎分为负责 prefill 的 prefill 引擎 (PE) 和负责 decode 的 decode 引擎 (DE).

• Traffic Manager (§5). Each engine contains a traffic manager to conduct (1) Host-Device memory copies (H2D & D2H), (2) KV-Cache transfers between PEs and DEs, and (3) KV-Cache reads/writes from/to storage via the storage NIC. We adopt a CNIC-centric traffic management approach, detailed in §5, to prevent KV-Cache traffic from affecting communications in model inference.

• 流量管理器 (§5). 每个引擎内含一个流量管理器, 负责 (1) 主机与设备间的内存拷贝 (H2D 与 D2H), (2) PE 与 DE 之间的 KV-Cache 传输, (3) 经存储网卡对存储读写 KV-Cache. 我们采用以 CNIC 为中心的流量管理方式 (详见 §5), 防止 KV-Cache 流量影响模型推理中的通信.

• Request Scheduler (§6). A central scheduler that receives client requests and distributes them across engines. It is also responsible for dynamically distributing data traffic between two paths (Figure 4).

• 请求调度器 (§6). 一个中心调度器, 接收客户端请求并把请求分发到各引擎. 它还负责在两条路径之间动态分配数据流量 (图 4).

4.1 Dual-Path Loading · 双路径加载

In addition to the conventional storage-to-prefill path, DualPath introduces a novel storage-to-decode path, allowing KV-Cache to be loaded first into a decode engine and then transferred to the prefill engine via high-bandwidth RDMA over the compute network. By dynamically distributing load across both paths, the system aggregates the storage NIC

Figure 4: Dual-path loading illustration. The scheduler dynamically distributes data traffic between the two paths.

bandwidth of all engines — including otherwise-idle decode side NICs — and eliminates the asymmetric bandwidth saturation that limits existing systems. This approach transforms the storage I/O from a single-bottleneck resource into a glob ally pooled and schedulable capacity. The exact data flows of the dual-path are described below.

除了传统的「存储到 prefill」路径, DualPath 引入一条新的「存储到 decode」路径: KV-Cache 先加载进某个 decode 引擎, 再经计算网络用高带宽 RDMA 传给 prefill 引擎. 通过在两条路径之间动态分配负载, 系统把所有引擎的存储网卡带宽 (包括原本闲着的 decode 侧网卡) 汇总起来, 消除限制现有系统的不对称带宽饱和. 这种做法把存储 I/O 从一个单点瓶颈资源, 变成全局汇总, 可调度的容量. 双路径的具体数据流如下.

To implement dual-path loading, DualPath allocates a small amount of DRAM as buffers on each PE and DE, called PE buffer and DE Buffer.

为实现双路径加载, DualPath 在每个 PE 和 DE 上分配少量 DRAM 作缓冲区, 称为 PE buffer 和 DE buffer.

Prefill PE read path. First, the KV-Caches of hit tokens are read from persistent storage into the PE buffer (as Label 1 and 2 shown in Figure 4a). Before the computation of an attention layer, those KV-Caches of that layer are transferred to PE HBM (3 and 4) to compute the KV-Cache of cache-miss prompt tokens. Then, all KV-Caches of both hit and miss tokens are transferred to the DE buffer to form the complete prompt KV-Cache (5-7). This process (3-7) repeats ^𝑛𝑙𝑎𝑦𝑒𝑟 times. During the prefill forward pass, transfers overlap with computation.

Prefill 阶段的 PE 读路径. 先, 命中 token 的 KV-Cache 从持久存储读入 PE buffer (图 4a 的标号 1 和 2). 在计算某个注意力层之前, 把这一层的 KV-Cache 传到 PE 的 HBM (3 和 4), 用来计算未命中的 prompt token 的 KV-Cache. 然后把命中与未命中 token 的全部 KV-Cache 传到 DE buffer, 拼成完整的 prompt KV-Cache (5-7). 过程 (3-7) 重复 $n_{layer}$ 次. prefill 前向过程中, 传输与计算重叠.

Prefill DE read path. The KV-Caches of hit tokens are first read into DE buffer (as Label 1 and 2 shown in Figure 4b). During PE prefill, KV-Cache for the corresponding layer is read from the DE buffer, also overlapping with computation (3-5). This process repeats $n _ { l a y e r }$ times. After a layer’s computation completes, only the KV-Caches of miss tokens are transferred to DE buffer and merged with the existing hit token KV-Cache.

Prefill 阶段的 DE 读路径. 命中 token 的 KV-Cache 先读入 DE buffer (图 4b 的标号 1 和 2). PE 做 prefill 时, 从 DE buffer 读取对应层的 KV-Cache, 同样与计算重叠 (3-5). 这一过程重复 $n_{layer}$ 次. 一层计算完成后, 只把未命中 token 的 KV-Cache 传到 DE buffer, 与已有的命中 token KV-Cache 合并.

Decode Phase. After receiving the complete prompt KV-Cache in DE buffer (including loaded KV-Cache via PE read path and the KV-Cache of newly appended tokens), the decode phase begins. The DE first allocates HBM and performs host-to-device (H2D) transfers (Label 8 and 9 in Figure 4a; Label 6 and 7 in Figure 4b), then releases CPU memory before starting decode. The design of DE buffer imposes bandwidth pressure on DRAM and CNIC (an extra H2D), which could

be avoided by directly bypassing it via GPU Direct RDMA. However, since the generation length is typically short in this scenario, time-to-first-token (TTFT) accounts for a non-negligible portion of the total end-to-end request time. Introducing DE buffer helps reduce GPU memory usage. During decode, whenever a full block of tokens (e.g., 64 tokens) is accumulated, it is immediately persisted to disk.

Decode 阶段. DE buffer 收齐完整的 prompt KV-Cache (包括经 PE 读路径加载的 KV-Cache 和新追加 token 的 KV-Cache) 后, decode 阶段开始. DE 先分配 HBM 并做主机到设备 (H2D) 的传输 (图 4a 的标号 8 和 9; 图 4b 的标号 6 和 7), 然后释放 CPU 内存, 再开始 decode. DE buffer 的设计给 DRAM 和 CNIC 增加了带宽压力 (多一次 H2D), 用 GPUDirect RDMA 直接绕过它本可以避免. 但这一场景里生成长度通常很短, TTFT 在请求端到端总时间里占不小的比例, 引入 DE buffer 有助于减少 GPU 显存占用. decode 期间, 每攒满一个 token 块 (例如 64 个 token), 就立即持久化到磁盘.

Different Block Layouts. We adopt two different block layouts: Full Block and Layer Block, which contain all layers and a single layer, respectively. Detailed layout can be found in §A.5. For all interactions with storage, we adopt Full Blocks. In the PE read case, KV-Cache loading to PE HBM and transfer to DE Buffer occur in a layerwise streaming fashion, both using Layer Blocks. Similarly, for the DE read path, transfers from the DE Buffer to the PE HBM use Layer Blocks.

不同的块布局. 我们采用两种块布局: Full Block 和 Layer Block, 分别包含全部层和单独一层. 具体布局见 §A.5. 与存储的所有交互都用 Full Block. 在 PE 读路径中, KV-Cache 加载到 PE HBM 和传到 DE buffer 都以逐层流式方式进行, 都用 Layer Block. 同样, DE 读路径中从 DE buffer 到 PE HBM 的传输也用 Layer Block.

4.2 Bottleneck-Free Analysis · 无瓶颈分析

We demonstrate that the system can fully saturate all storage NICs without introducing compute-NIC or DRAM bottlenecks, under most reasonable P/D ratios. We assume a well-configured PCIe topology (each pair of GPU–NIC is under the same PCIe switch), load-balanced task scheduling, no congestion on the computation network, and that storage read bandwidth is fully utilized.

我们说明, 在多数合理的 P/D 比例下, 系统可以把所有存储网卡跑满, 而不在计算网卡或 DRAM 上引入新瓶颈. 假设: PCIe 拓扑配置良好 (每对 GPU 与网卡挂在同一个 PCIe 交换机下), 任务调度负载均衡, 计算网络无拥塞, 存储读带宽被充分利用.

Notation. Let 𝑃 and 𝐷 denote the number of prefill and decode nodes, respectively. Each node has 𝑔 GPUs, each with one compute NIC of bandwidth 𝐵. The storage bandwidth per machine is 𝑠 × 𝐵 (shared by all engines on that machine); 𝑀 is the memory bandwidth per machine.

记号. 令 $P$ 和 $D$ 分别表示 prefill 节点数和 decode 节点数. 每个节点有 $g$ 块 GPU, 每块 GPU 配一块带宽为 $B$ 的计算网卡. 每台机器的存储带宽为 $s\times B$ (由该机器上所有引擎共享); $M$ 是每台机器的内存带宽.

Traffic per PE-DE pair. We assume that the storage read bandwidth is fully utilized and that task scheduling is loadbalanced. Under load balancing, storage NIC bandwidth is evenly shared. The traffic per pair for the PE read path (all steps in Figure 4a) is $T _ { p } = B s / ( D g ^ { 2 } )$ ; for the DE read path

(Figure 4b) it is $T _ { c } = B s / ( P g ^ { 2 } )$ . Link traffic is the sum over all pairs using that link.

每对 PE-DE 的流量. 假设存储读带宽被充分利用, 任务调度负载均衡. 负载均衡时, 存储网卡带宽被均匀分摊. PE 读路径 (图 4a 的所有步骤) 每对的流量为 $T_p=Bs/(Dg^2)$; DE 读路径 (图 4b) 每对的流量为 $T_c=Bs/(Pg^2)$. 一条链路上的流量是使用该链路的所有 PE-DE 对的流量之和.

PE CNIC Bandwidth Analysis. For PE CNIC, loopback traffic (i.e., H2D and D2H that does not traverse switches) exists, so the total traffic on the PCIe side is always greater than or equal to the switch-direction traffic, regardless of read or write operations. Therefore, we only need to compute the pressure on the PCIe side. Read operations include PE paths (3) and (5), with total traffic over all pairs:

PE 的 CNIC 带宽分析. PE 的 CNIC 上存在回环流量 (即不经过交换机的 H2D 和 D2H), 所以不论读还是写, PCIe 一侧的总流量总是大于等于交换机方向的流量. 因此只需计算 PCIe 一侧的压力. 读操作包括 PE 路径的 (3) 和 (5), 所有 PE-DE 对的总流量为:

Since $s \leq g$ always holds in practice, the read direction is always bottleneck-free. Write operations include PE path (4) and DE path (5), with total traffic:

由于实际中 $s\le g$ 总是成立, 读方向永远不会成为瓶颈. 写操作包括 PE 路径的 (4) 和 DE 路径的 (5), 总流量为:

Then, we obtain:

由此得到:

DE CNIC Bandwidth Analysis. For DE CNIC, read operations include PE path 8 and DE paths 3/6, with traffic:

DE 的 CNIC 带宽分析. 对 DE 的 CNIC, 读操作包括 PE 路径的 8 和 DE 路径的 3 与 6, 流量为:

Then, we obtain:

由此得到:

Write operations include PE paths 7/9 and DE path $^ { 7 , }$ with traffic:

写操作包括 PE 路径的 7 与 9, 以及 DE 路径的 7, 流量为:

This implies:

这意味着:

DRAM Pressure Analysis. DRAM is half-duplex, so we sum the read and write pressures. For PE MEM, the pressure is 2𝑠𝐵, which generally does not exceed memory bandwidth. For DE MEM, following the similar analysis above, we can get the pressure is $\left( 3 + 2 P / D \right)$ 𝐵𝑠. Requiring the DE MEM pressure to be less than or equal to $M ,$ we obtain:

DRAM 压力分析. DRAM 是半双工的, 所以把读写压力相加. PE 内存的压力是 $2sB$, 一般不会超过内存带宽. 对 DE 内存, 按与上面类似的分析, 压力为 $(3+2P/D)Bs$. 要求 DE 内存压力不超过 $M$, 得到:

Summary. Combining all the above analyses, we have:

小结. 综合以上分析, 有:

For $( g = 8 , s = 1 )$ with 𝑀 ≈ 500 GB/s and $B s \approx 5 0 ; \mathrm { G B / s } ,$ , the bottleneck-free range is $\textstyle { \frac { 1 } { 7 } } \leq \operatorname { P } / \operatorname { D } \leq { \frac { 7 } { 2 } }$ , which covers most practical configurations.

取 $g=8$, $s=1$, $M\approx 500$ GB/s, $Bs\approx 50$ GB/s, 无瓶颈区间为 $1/7\le P/D\le 7/2$, 覆盖了多数实际配置.

拆开: 式 (4) 里 $T_c$ 为什么乘 2, 式 (6) 里 $T_p$ 为什么乘 2? 各项对应图 4 的哪根箭头? 答: 按图 4 的编号逐项对. 先看 PE 的 CNIC: 式 (1) 的系数 2 来自同一份命中 KV 在 PE 的 CNIC 上被读两次, (3) 从 PE buffer 读出做 H2D, (5) 从 PE 的 HBM 读出发往 DE; 式 (1) 化简后是 $s\le g/2$, 比正文写的 $s\le g$ 更严, 实验的 $g=8$, $s=1$ 离这个界很远. DE 的 CNIC 读方向: PE 路径只有 (8), 即把 DE buffer 里已拼好的 KV 读出做 H2D, 计一次 $T_p$; DE 路径有 (3) 把 DE buffer 里的命中 KV 读出发给 PE, 和 (6) 读出做 H2D, 同一份数据读两次, 所以 $2T_c$. DE 的 CNIC 写方向: PE 路径有 (7) 把从 PE 收到的 KV 写进 DE buffer, 和 (9) H2D 写进 DE 的 HBM, 计 $2T_p$; DE 路径只有 (7) H2D 写进 HBM, 计 $T_c$ (未命中 token 的回传量相对命中部分可以忽略, 式中没计). 每块 DE GPU 与 $Pg$ 块 PE GPU 成对, 乘上 $Pg$ 即得式 (4) 和式 (6). 代入 $g=8$, $s=1$: 式 (5) 给 6, 式 (7) 给 3.5, 式 (8) 给 $(10-3)/2=3.5$, 上界由 DE 的 CNIC 写方向和 DE 内存同时卡在 3.5. 这些结论都建立在两侧存储网卡同时读满, 请求均匀分到所有 PE-DE 对上的假设之上.

4.3 Practical Challenges · 实际挑战

The dual-path architecture fundamentally reorients data movement: KV-Cache can be loaded either directly from storage into prefill engines or indirectly via decode engines, thereby aggregating storage bandwidth across all engines and breaking the prefill-side I/O bottleneck. However, realizing this high-level design in a practical system introduces three interrelated challenges. We briefly outline these challenges below and refer the reader to the corresponding sections for details.

双路径架构从根本上改变了数据移动的方向: KV-Cache 既可以从存储直接加载进 prefill 引擎, 也可以经 decode 引擎间接加载, 从而汇总所有引擎的存储带宽, 打破 prefill 侧的 I/O 瓶颈. 但在实际系统里实现这一高层设计, 会遇到三个相互关联的挑战. 下面简述, 细节见对应章节.

Fine-grained data transfer (§5). The layer-wise execution paradigm, while essential for overcoming HBM capacity limits, fragments the KV-Cache into numerous fine-grained blocks [37]. Transferring this multitude of fine-grained data chunks between storage, host DRAM, and GPU HBM must incur minimal overhead and seamlessly overlap with computation to realize throughput gains.

细粒度数据传输 (§5). 逐层执行的方式对克服 HBM 容量限制必不可少, 但它把 KV-Cache 切成大量细粒度的块 [37]. 要拿到吞吐收益, 在存储, 主机 DRAM 和 GPU HBM 之间搬运这些大量细碎数据块时, 开销必须很小, 并且要与计算无缝重叠.

Traffic isolation (§5). The complex data path in DualPath introduces additional KV-Cache transfer traffic on both the compute network and PCIe links. A primary concern is that this traffic may interfere with existing latency-sensitive collective communication operations essential for model execution — such as AllToAll in expert parallel [56] and ReduceScatter/AllGather in tensor/context parallel. Since these collective communications are critical to end-to-end inference latency, a key challenge lies in exploiting spare I/O bandwidth without degrading model inference performance.

流量隔离 (§5). DualPath 的数据通路在计算网络和 PCIe 链路上增加了 KV-Cache 传输流量. 这些流量可能干扰对延迟敏感的模型集合通信, 包括专家并行的 AllToAll [56]、张量并行或上下文并行的 ReduceScatter/AllGather, 进而拉长端到端推理时间. 因此系统需要利用空闲 I/O 带宽, 同时避免影响模型通信.

Dynamic load balancing (§6). As we are adopting two different paths for KV-cache loading, the system must promptly decide which path to use for each request. A naive policy could overload one path, recreating the original bottleneck. The traffic scheduler must balance multiple factors in realtime: storage NIC queue lengths, computational load on GPUs, and request workload characteristics.

动态负载均衡 (§6). 既然 KV-Cache 加载有两条路径, 系统就必须及时为每个请求决定走哪条. 朴素的策略可能让某一条路径过载, 重新造出原来的瓶颈. 流量调度器必须实时权衡多个因素: 存储网卡队列长度, GPU 上的计算负载, 以及请求本身的负载特征.

5 CNIC-CENTRIC TRAFFIC MANAGER · 以 CNIC 为中心的流量管理器

Modern LLM inference systems employ a range of advanced data transfer technologies — such as on-chip CUDA copy engine and GPUDirect Storage [34] — to move data efficiently between storage, host memory, and GPU HBM. However, all these mechanisms can interfere with latency-sensitive collective communications (e.g., EP AllToAll) during model execution. This arises for two primary reasons: (1) such transfer technologies often operate over separate paths that do not share the same QoS controls as the compute network, and (2) existing GPUs do not support PCIe QoS [39], making it difficult to shield model inference communication from other traffic contending for PCIe bandwidth. Additionally, because the collective communications occur in rapid, sub-millisecond-level bursts, it is impractical to rely on a

software-based traffic shaper to interleave lower-priority I/O operations between these high-priority traffic windows.

现代 LLM 推理系统用多种先进的数据传输技术 (例如片上 CUDA copy engine 和 GPUDirect Storage [34]) 在存储, 主机内存与 GPU HBM 之间高效搬运数据. 但这些机制都可能干扰模型执行期间对延迟敏感的集合通信 (例如 EP 的 AllToAll). 原因主要有两个: (1) 这些传输技术往往走独立的通路, 与计算网络不共享同一套 QoS 控制; (2) 现有 GPU 不支持 PCIe QoS [39], 很难把模型推理通信与其他争抢 PCIe 带宽的流量隔开. 此外, 集合通信以亚毫秒级的快速突发出现, 靠软件流量整形器把低优先级 I/O 插进这些高优先级流量窗口之间的空隙, 并不现实.

To address this, we propose a CNIC–centric data transfer approach which is widely adopted in our production deployment: all data traffic in or out of a GPU, including local H2D/D2H copy, must go through the GPU’s paired CNIC with a GPUDirect RDMA [33] data path. By consolidating all traffic onto the compute network, we can leverage the native QoS capabilities of compute network to enforce strict traffic differentiation.

为此我们提出以 CNIC 为中心的数据传输方式, 它已在我们的生产部署中广泛使用: 进出 GPU 的所有数据流量, 包括本机的 H2D/D2H 拷贝, 都必须经该 GPU 配对的 CNIC, 走 GPUDirect RDMA [33] 数据通路. 把全部流量收拢到计算网络上, 就能借计算网络原生的 QoS 能力强制做严格的流量区分.

5.1 Traffic Isolation · 流量隔离

For the InfiniBand-based network, we leverage virtual lanes (VLs) [5] to enforce isolation between different traffic classes. All model inference communication traffic is assigned to a dedicated high-priority VL, while all other traffic, including KV-Cache transfer, is mapped to a separate low-priority VL. We configure the VL arbiters of all network switches and NICs with a weighted round-robin policy that reserves approximately 99% of total bandwidth to high-priority VL. The remaining bandwidth is allocated to the low-priority VL to prevent starvation. This configuration ensures that model execution traffic is virtually unaffected by KV-cache trans fers, while still allowing KV-cache traffic to opportunistically utilize otherwise idle bandwidth in the compute network. Detailed configurations are described in §A.1.

对基于 InfiniBand 的网络, 我们用虚拟通道 (VL) [5] 在不同流量类别之间强制隔离. 所有模型推理通信流量分到一条专用的高优先级 VL, 其余流量 (包括 KV-Cache 传输) 映射到另一条低优先级 VL. 所有交换机和网卡的 VL 仲裁器都配成加权轮询, 给高优先级 VL 预留约 99% 的总带宽, 剩下的带宽分给低优先级 VL, 防止饿死. 这样配置后, 模型执行流量几乎不受 KV-Cache 传输影响, 而 KV-Cache 流量仍能见缝插针地使用计算网络里原本空闲的带宽. 详细配置见 §A.1.

Although our experiments are conducted on an InfiniBandbased network, the same design principles naturally extend to other interconnect technologies. DualPath can be conducted on RDMA over Converged Ethernet (RoCE) by leveraging Traffic Class (TC) and Differentiated Services Code Point (DSCP) markings [6, 20] in conjunction with hardware packet queues. Emerging technologies such as UnifiedBus [1] and Ultra Ethernet [9] are likewise converging on QoS mechanisms for heterogeneous traffic, which can directly support the requirements of DualPath.

我们的实验在 InfiniBand 网络上进行, 但同样的设计原则可以自然推广到其他互连技术. 在 RoCE (RDMA over Converged Ethernet) 上, DualPath 可以借 Traffic Class (TC) 与 DSCP (Differentiated Services Code Point) 标记 [6, 20], 配合硬件包队列来实现. UnifiedBus [1] 和 Ultra Ethernet [9] 等新兴技术也在向面向异构流量的 QoS 机制收敛, 可以直接满足 DualPath 的需求.

5.2 CNIC-Assisted KV-Cache Copy · CNIC 辅助的 KV-Cache 拷贝

Existing GPU data transfer technologies include GPUDirect Storage [34], which loads KV-Cache from storage backend to GPU HBM, and CUDA copy engine, which directly copies host DRAM to GPU via PCIe. However, these methods fail to isolate KV-Cache traffic from high-priority latency-sensitive collective communications in model execution, severely degrading inference performance.

现有的 GPU 数据传输技术包括 GPUDirect Storage [34] (把 KV-Cache 从存储后端加载到 GPU HBM) 和 CUDA copy engine (经 PCIe 把主机 DRAM 直接拷到 GPU). 但这些方法无法把 KV-Cache 流量与模型执行中高优先级, 延迟敏感的集合通信隔开, 会严重拖累推理性能.

To solve the limitations of existing approaches, we adopt a CNIC-assisted H2D/D2H data path. For KV-Cache loading, we first read the KV-Cache into host DRAM from the storage backend. Then, we submit an RDMA Write request to the GPU’s paired CNIC to perform local H2D copy. Storing newly generated KV-Cache follows a symmetric process: it

is first transferred to host DRAM via CNIC, then persisted to the storage backend over the storage network. This design establishes the CNIC as the central QoS scheduler for all GPU PCIe traffic, allowing its VL arbiter to prioritize the inference communication traffic and perform KV-Cache transfer using spare PCIe bandwidth.

为解决现有方法的局限, 我们采用 CNIC 辅助的 H2D/D2H 数据通路. 加载 KV-Cache 时, 先从存储后端把 KV-Cache 读进主机 DRAM, 再向该 GPU 配对的 CNIC 提交一个 RDMA Write 请求, 完成本机的 H2D 拷贝. 存储新生成的 KV-Cache 是对称的过程: 先经 CNIC 传到主机 DRAM, 再经存储网络持久化到存储后端. 这一设计让 CNIC 成为该 GPU 全部 PCIe 流量的中心 QoS 调度点, 由它的 VL 仲裁器优先放行推理通信流量, 用空闲的 PCIe 带宽做 KV-Cache 传输.

Although this approach may appear to take a detour compared to GPUDirect Storage (which directly reads KV-Cache to GPU HBM) and CUDA copy engine (which directly copies host memory to GPU HBM), to our best knowledge, this is currently the only practical method to ensure that KV-Cache load/store does not degrade the performance of critical model–execution communication.

与 GPUDirect Storage (直接把 KV-Cache 读进 GPU HBM) 和 CUDA copy engine (直接把主机内存拷进 GPU HBM) 相比, 这种做法看上去绕了路. 但据我们所知, 它是目前唯一可行的方法, 能保证 KV-Cache 的加载与存储不拖累关键的模型执行通信.

We also observe that CNIC-assisted H2D and D2H outperform the CUDA copy engine when handling a large number of small data chunks. Our measurements show that submitting a single copy operation via cudaMemcpyAsync incurs a latency overhead of approximately $5 - 7 \mu s$ . We failed to further break down this overhead due to the closed-source nature of CUDA driver. In contrast, submitting one RDMA Write work request involves only a few mmio writes to NIC registers in user space and takes only around 1𝜇𝑠. Furthermore, the RDMA work submission overhead can be significantly amortized by leveraging doorbell batching [25].

我们还观察到, 处理大量小数据块时, CNIC 辅助的 H2D 和 D2H 比 CUDA copy engine 更快. 测量显示, 用 cudaMemcpyAsync 提交一次拷贝约有 5-7 $\mu s$ 的延迟开销; 由于 CUDA 驱动闭源, 我们没能进一步拆解这部分开销. 相比之下, 提交一个 RDMA Write 工作请求只需在用户态对网卡寄存器做几次 mmio 写, 约 1 $\mu s$. 而且借助 doorbell batching [25], RDMA 的提交开销还能被大幅摊薄.

6 ADAPTIVE REQUEST SCHEDULER · 自适应请求调度器

Although our theoretical analysis shows promising results, imbalanced load reduces hardware utilization, and in this scenario, we need to consider two dimensions of balance simultaneously: (1) NIC traffic, and (2) the utilization balance of GPUs. We divide scheduling into two levels: inter-engine scheduling, which assigns requests to a (PE, DE) pair and selects the read path (PE or DE) for each request; and intraengine scheduling, which determines which requests are included in each forward batch for computation.

理论分析给出了乐观的结果, 但负载不均会降低硬件利用率. 在这一场景下需要同时考虑两个维度的均衡: (1) 网卡流量; (2) GPU 利用率. 我们把调度分为两级: 引擎间调度, 把请求分配给一个 (PE, DE) 对, 并为每个请求选择读路径 (PE 或 DE); 引擎内调度, 决定每个前向 batch 里包含哪些请求参与计算.

6.1 Inter-Engine Scheduling · 引擎间调度

We organize engines into groups to reduce the scheduler pressure. Only the engine rank 0, called Leader Engine, interacts with the scheduler. All engines of a group are all PEs or all DEs. All engines on one node are guaranteed to be in the same group. All engines in one group proactively fetch tasks together regularly. When fetching new requests, each engine 𝑒 reports (1) $s e q _ { e }$ , the number of requests assigned to it that have not yet completed; (2) the total token count 𝑡𝑜𝑘_𝑒 over those 𝑠𝑒𝑞_𝑒 requests; and (3) the disk reading queue length 𝑟𝑒𝑎𝑑 $\mathcal { A } n ( e )$ of the node 𝑛(𝑒) that engine 𝑒 belongs to. GPU load, disk read load, and network load are all strongly correlated with token count. We therefore use token count as a proxy and aim to balance it across engines.

为减轻调度器压力, 我们把引擎分组. 只有 rank 0 的引擎 (称为 Leader Engine) 与调度器交互. 同一组的引擎要么全是 PE, 要么全是 DE; 同一节点上的所有引擎保证在同一组. 一组里的所有引擎定期一起主动拉取任务. 拉取新请求时, 每个引擎 $e$ 上报: (1) $seq_e$, 已分配给它但尚未完成的请求数; (2) 这 $seq_e$ 个请求的 token 总数 $tok_e$; (3) 引擎 $e$ 所在节点 $n(e)$ 的磁盘读取队列长度 $read_q$. GPU 负载, 磁盘读负载和网络负载都与 token 数强相关, 所以我们用 token 数作代理指标, 目标是让各引擎的 token 数均衡.

Figure 5: An illustration of Inter-Engine PE Scheduling. All eight GPUs are in the same PE engine group and the scheduler will choose the best.

PE Scheduling. All requests arriving at the scheduler enter a waiting queue and are scheduled in a FIFO order. The scheduling algorithm is invoked when a PE group initiates a fetch request. An illustration of the inter-engine scheduling process is shown in Figure 5. We define two constants, short reading queue threshold 𝛼, and unfinished token upper limit $\beta ,$ measured in tokens. All engines are split into three categories: (1) overloaded engines where $t o k _ { e } > \beta ; ( 2 )$ engines on nodes with short disk reading queues where 𝑟𝑒𝑎𝑑 $_ q _ { n ( e ) } \leq \alpha$