arXiv:2408.14158v2 [cs.DC] 31 Aug 2024

Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning · 面向深度学习的高性价比软硬件协同设计

Wei An, Xiao Bi, Guanting Chen, Shanhuang Chen, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Wenjun Gao, Kang Guan, Jianzhong Guo, Yongqiang Guo, Zhe Fu, Ying He, Panpan Huang, Jiashi Li,

Wenjun Gao, Kang Guan, Jianzhong Guo, Yongqiang Guo, Zhe Fu, Ying He, Panpan Huang, Jiashi Li, Wenfeng Liang, Xiaodong Liu, Xin Liu, Yiyuan Liu, Yuxuan Liu, Shanghao Lu, Xuan Lu, Xiaotao Nie, Tian Pei, Junjie Qiu, Hui Qu, Zehui Ren, Zhangli Sha, Xuecheng Su, Xiaowen Sun, Yixuan Tan, Minghui Tang, Shiyu Wang, Yaohui Wang, Yongji Wang, Ziwei Xie, Yiliang Xiong, Yanhong Xu, Shengfeng Ye, Shuiping Yu, Yukun Zha, Liyue Zhang*, Haowei Zhang, Mingchuan Zhang, Wentao Zhang, Yichao Zhang, Chenggang Zhao, Yao Zhao, Shangyan Zhou, Shunfeng Zhou, Yuheng Zou

DeepSeek-AI, Beijing, China

research@deepseek.com

Abstract—The rapid progress in Deep Learning (DL) and Large Language Models (LLMs) has exponentially increased demands of computational power and bandwidth. This, combined with the high costs of faster computing chips and interconnects, has significantly inflated High Performance Computing (HPC) construction costs. To address these challenges, we introduce the Fire-Flyer AI-HPC architecture, a synergistic hardware-software co-design framework and its best practices. For DL training, we deployed the Fire-Flyer 2 with 10,000 PCIe A100 GPUs, achieved performance approximating the DGX-A100 while reducing costs by half and energy consumption by 40%. We specifically engineered HFReduce to accelerate allreduce communication and implemented numerous measures to keep our Computation-Storage Integrated Network congestion-free. Through our software stack, including HaiScale, 3FS, and HAI-Platform, we achieved substantial scalability by overlapping computation and communication. Our system-oriented experience from DL training provides valuable insights to drive future advancements in AI-HPC.

深度学习和大语言模型发展很快, 对算力和带宽的需求呈指数增长. 更快的计算芯片和互连又很贵, HPC 集群的建设成本因此被大幅推高. 为此我们提出 Fire-Flyer AI-HPC 架构, 这是一套软硬件协同设计的框架和相应的最佳实践. 在深度学习训练上, 我们部署了配备 10,000 张 PCIe A100 的 Fire-Flyer 2, 性能接近 DGX-A100, 成本降低一半, 能耗降低 40%. 我们专门设计了 HFReduce 来加速 allreduce 通信, 并采取多项措施让计算与存储一体化网络不发生拥塞. 借助 HaiScale, 3FS 和 HAI Platform 组成的软件栈, 我们通过计算与通信重叠取得了良好的扩展性. 这些来自深度学习训练的系统经验, 可以为 AI-HPC 后续的发展提供参考.

Index Terms—High Performance Computing, Cost-Effective, All-Reduce, Best Practices, Deep Learning, Machine Learning, Large Language Models, Artificial Intelligence Infrastructure

关键词: 高性能计算, 高性价比, All-Reduce, 最佳实践, 深度学习, 机器学习, 大语言模型, 人工智能基础设施.

I. INTRODUCTION

In recent years, Deep Learning (DL) [1] has developed rapidly and is widely used in image recognition, speech recognition, content generation, autonomous driving, and other areas. The rapid development of DL is fundamentally tied to the support rendered by data. Training with copious amounts of data demands massive computational resources. Relying on Moore's Law [2], computer speeds double every two years on average, but the pace of DL far exceeds this speed. In particular, Large Language Models (LLMs) [3]–[7] that have become popular in recent years have exploded the demand for computational resources and memory. The parameters of LLMs can reach tens to thousands of billions, requiring hundreds or thousands of GPUs for training. Although LLM training is challenging, the emerging capabilities resulting from more

parameters have shown the benefits of continued model expansion. Since then, researchers have gone down the path of making models bigger and never looked back. To acquire more computational resources, people have to expand more nodes. This leads to a surge in the cost of building AI infrastructure. How to reduce the cost of new data centers and how to build cost-effective clusters are also hot and challenging problems. Moreover, more nodes lead to higher energy consumption, which contradicts this era's goal of reducing carbon emissions and achieving carbon neutrality. Reducing energy consumption is also a challenging problem.

近年来深度学习 [1] 发展迅速, 广泛用于图像识别, 语音识别, 内容生成, 自动驾驶等领域. 它的快速发展离不开数据, 而用海量数据训练需要海量算力. 按摩尔定律 [2], 计算机速度平均每两年翻一番, 深度学习的发展速度远超于此. 近几年流行的大语言模型 [3]-[7] 让算力和内存需求急剧膨胀, 参数量可达数百亿到数万亿, 训练需要成百上千张 GPU. LLM 训练虽然困难, 参数增多带来的涌现能力证明了继续扩大模型的价值, 研究者从此沿着做大模型的路一直走下去. 要获得更多算力只能扩充节点, AI 基础设施的建设成本随之激增. 如何降低新数据中心的成本, 如何建设高性价比集群, 都是热门而困难的问题. 节点越多能耗越高, 这与当下减排和碳中和的目标相冲突, 降低能耗同样是难题.

In this paper, leverage our practical experience accumulated over the years to propose cost-effective strategies cost-effective strategies for constructing AI-HPC systems suitable for deep learning and LLMs.

本文基于我们多年积累的实践经验, 提出适合深度学习和 LLM 的 AI-HPC 系统高性价比建设策略.

Fire-Flyer AI-HPC Architecture: We have deployed a cluster composed of 10,000 PCIe A100 GPUs for Deep Learning training purposes. Details about the GPU nodes and network topology are provided in Section III-A, where we compare our architecture to the NVIDIA DGX-A100 [8] in terms of cost-effectiveness and lower $CO_{2}$ emissions. In contrast, we must invest more in software optimizations to address the performance challenges of the PCIe architecture. The following sections will discuss about software-hardware co-design.

Fire-Flyer AI-HPC 架构: 我们部署了由 10,000 张 PCIe A100 组成的深度学习训练集群. GPU 节点和网络拓扑的细节见 Section III-A, 那里会从性价比和 $CO_2$ 排放两方面与 NVIDIA DGX-A100 [8] 比较. 代价是我们必须在软件优化上投入更多, 以弥补 PCIe 架构的性能短板. 后续章节讨论软硬件协同设计.

Key Technical Topics in our Architecture · 本架构的关键技术点

- Network Co-Design: The Two-Layer Fat-Tree Network [9] integrates storage and computation network, as shown in Section III-B. The entire network is divided into two zones, and the platform supports cross-zone tasks. To prevent congestion, we employed various network tunings detailed in Section VI-A.

• 网络协同设计: 两层 Fat-Tree 网络 [9] 把存储网络和计算网络合为一张, 见 Section III-B. 整个网络分为两个区, 平台支持跨区任务. 为防止拥塞, 我们做了多项网络调优, 见 Section VI-A.

- HFReduce: Achieves computation-communication overlap via asynchronous allreduce on the CPU, outperforming NVIDIA Collective Communications Library (NCCL) [10] on our PCIe architecture, as discussed in Section IV.

• HFReduce: 在 CPU 上做异步 allreduce, 实现计算与通信重叠, 在我们的 PCIe 架构上性能超过 NVIDIA 集合通信库 NCCL [10], 见 Section IV.

Authors are listed in alphabetical order of their surnames.

作者按姓氏字母顺序排列.

SC24, November 17-22, 2024, Atlanta, Georgia, USA 979-8-3503-5291-7/24/$31.00 ©2024 IEEE

- HaiScale: As described in Section V, optimizes parallelism methods for our PCIe architecture, such as Data Parallelism (DP), Pipeline Parallelism (PP) [11] [12], Tensor Parallelism (TP) [13] [14], Experts Parallelism (EP) [15]–[17], Fully Sharded Data Parallel(FSDP) [18] and Zero Redundancy Optimizer(ZeRO) [19].

• HaiScale: 见 Section V, 针对 PCIe 架构优化了多种并行方式, 包括数据并行 (DP), 流水线并行 (PP) [11] [12], 张量并行 (TP) [13] [14], 专家并行 (EP) [15]-[17], 全分片数据并行 (FSDP) [18] 和零冗余优化器 (ZeRO) [19].

- 3FS Distributed File System: Addresses I/O bottlenecks in big data AI tasks, configures with our communication and network tuning, reduces congestion in storage and computation integrated network topology, as detailed in Section VI-B.

• 3FS 分布式文件系统: 解决大数据 AI 任务的 I/O 瓶颈, 与我们的通信和网络调优配合, 减少存算一体网络拓扑中的拥塞, 见 Section VI-B.

- HAI Platform [20]: Offers task scheduling, fault handling, and disaster recovery, enhancing utilization and reducing costs. It provides an open-box solution for deep-learning researchers. It is already open-sourced: https://github.com/HFAiLab/hai-platform

• HAI Platform [20]: 提供任务调度, 故障处理和容灾, 提高利用率, 降低成本, 为深度学习研究者提供开箱即用的方案. 已开源: https://github.com/HFAiLab/hai-platform

Stability and Robustness: These are crucial topics in HPC. Our systems are equipped with robust mechanisms to handle hardware failures, minimizing downtime and impact on operations. These mechanisms, discussed in Section VII, include:

稳定性与健壮性: 这是 HPC 的关键问题. 我们的系统配有应对硬件故障的机制, 尽量减少停机时间和对业务的影响. 这些机制在 Section VII 讨论, 包括:

- Disaster recovery through our checkpoint manager

• 通过 checkpoint 管理器实现容灾

- A validator utility for detecting hardware failures

• 用于发现硬件故障的 validator 工具

- An overview of real hardware failure data from our cluster over the past year.

• 我们集群过去一年真实硬件故障数据的概览.

We hope these insights will be beneficial to industry peers and researchers alike.

希望这些经验对业界同行和研究者都有帮助.

Discussion and Future Work: In Section VIII, we address some common questions regarding PCIe architecture, such as congestion control, maintenance cost, and stability compared with other architectures. In Section IX, we propose the next generation of PCIe architecture, which is aimed at Mixture-of-Experts Large Language Models training and primarily utilizes multi-NICs and a Multi-Plane network.

讨论与未来工作: Section VIII 回答关于 PCIe 架构的一些常见问题, 例如拥塞控制, 维护成本, 以及与其他架构相比的稳定性. Section IX 提出面向 MoE 大语言模型训练的下一代 PCIe 架构, 主要采用多网卡和多平面网络.

II. BACKGROUND · 背景

A. Evolution of Deep Learning · 深度学习的演进

The revolution in Machine Learning and Deep Learning began in 2012 with AlexNet [21], which outperformed traditional methods in image classification, marking the onset of big data utilization and increased computational demands. The emergence of ResNet [22], with its deeper layers, further broadened the horizons of image processing, truly bringing about the “deep” in “Deep Learning”. At the same time, big data-driven model training nudged the evolution of data storage technology, leading to the advent of all-flash SSD distributed file systems.

机器学习和深度学习的变革始于 2012 年的 AlexNet [21], 它在图像分类上超过了传统方法, 标志着大数据利用和算力需求增长的开端. ResNet [22] 层数更深, 进一步拓宽了图像处理的边界, 让深度学习真正「深」了起来. 同时, 大数据驱动的模型训练推动了存储技术演进, 全闪存 SSD 分布式文件系统随之出现.

Fast forward to 2017, Google's Transformer [23] made its grand entry, introducing the concept of "Attention is all you need", shaking up the field of Natural Language Processing (NLP). With the advent of more complex models like AlphaFold [24] and AlphaZero [25] highlighting the need for more computational power and memory, revealing the limitations of traditional FP64 / FP32 computing devices.

到 2017 年, Google 的 Transformer [23] 登场, 提出「Attention is all you need」, 改变了自然语言处理领域. AlphaFold [24], AlphaZero [25] 等更复杂的模型出现, 对算力和内存提出更高要求, 也暴露出传统 FP64/FP32 计算设备的局限.

Entering the 2020s saw the rise of LLMs as a game-changer in the AI sector. Research indicates that an upscale in the number of language model parameters and computational budget

Figure 1: The Exponential Growth of Computational Power for Deep Learning.

can significantly enhance model performance, given adequate training data. Consequently, despite requiring colossal computational resources, efforts are being made to train large models on tens or hundreds of billions or even trillions of parameters. Pioneering examples include GPT-3 [26] and PaLM [27], which occupy close to 1TB of GPU Memory. Recognizing the potential, industry giants have set up large AI clusters to train LLMs while constantly investing in computational power chips.

进入 2020 年代, LLM 成为 AI 领域的转折点. 研究表明, 只要训练数据充足, 增加语言模型参数量和计算预算可以显著提升模型性能. 因此尽管需要巨量算力, 人们仍在训练数百亿, 数千亿乃至万亿参数的大模型, 早期代表有 GPT-3 [26] 和 PaLM [27], 它们占用接近 1TB 显存. 看到这一潜力, 行业巨头纷纷建设大型 AI 集群训练 LLM, 并持续投资算力芯片.

The shift towards the Mixture-of-Experts (MoE) Models [28]–[30] architecture starting from GPT-4 [7], and the recent AI Generated Content (AIGC) multi-modal (Sora [31]) has amplified the demand for memory and computational resources. However, as AI development outpaces hardware development, leading to skyrocketing training costs, adopting cost-saving solutions has become imperative.

从 GPT-4 [7] 开始转向 MoE 架构 [28]-[30], 加上近来的 AIGC 多模态模型 (Sora [31]), 进一步放大了对内存和算力的需求. AI 发展快于硬件发展, 训练成本飙升, 采用省钱的方案已是必然.

Figure 1 illustrates the exponential growth of computational power for DL. And as summarized in Figure 2 [32], while AI's demand for computational power is growing at 10x per year, Moore's Law lags behind with hardware FLOPs increase at only 3.0x every two years, DRAM bandwidth at 1.6x, and interconnect bandwidth at 1.4x. This disparity necessitates more machines, raising DL training costs, particularly for LLM training, where the computational power required surpasses that of traditional HPC applications.

Figure 1 展示了深度学习算力需求的指数增长. Figure 2 [32] 总结了另一组趋势: AI 对算力的需求每年增长 10 倍, 而硬件 FLOPs 每两年只增长 3.0 倍, DRAM 带宽 1.6 倍, 互连带宽 1.4 倍. 这一差距意味着需要更多机器, 推高了深度学习训练成本, LLM 训练尤其如此, 其算力需求已超过传统 HPC 应用.

B. Challenges and Solutions in Models Training · 模型训练的挑战与对策

In Deep Learning training, a single task demands hundreds of GPUs and consumes substantial storage and network resources. This massive scale introduces system-level challenges:

深度学习训练中, 单个任务需要数百张 GPU, 并占用大量存储和网络资源. 这样的规模带来系统层面的挑战:

1. Efficiency: Firstly, achieving efficient training at this magnitude is crucial. Model FLOPs Utilization (MFU), which assesses the ratio of observed throughput to theoretical maximum throughput (assuming 100% peak FLOPS), serves as the standard metric for evaluating training efficiency. Training LLMs involves dividing models among GPUs that communicate extensively for progress. Besides communication, factors like operation optimization, data pre-processing, and GPU memory consumption significantly influence MFU. Multiple parallel strategies are employed to enhance efficiency:

2. 效率: 先要在这个量级上高效训练. 模型 FLOPs 利用率 (MFU) 是衡量训练效率的标准指标, 即实测吞吐与理论最大吞吐 (按 100% 峰值 FLOPS 计) 之比. LLM 训练要把模型切分到多张 GPU 上, GPU 之间需要大量通信. 除通信外, 算子优化, 数据预处理, 显存占用也会显著影响 MFU. 常用以下并行策略提高效率:

Figure 2: Scaling of Peak Hardware FLOPS, and Memory/Interconnect Bandwidth.

- Data Parallelism (DP): Models and optimizer states are replicated across multiple devices with data evenly distributed to all. For LLMs training, the Zero Redundancy Optimizer (ZeRO) [19] further enhances this method by sharding these states on each data parallel process and using allgather and reduce-scatter for parameter fetching and gradient calculation.

• 数据并行 (DP): 模型和优化器状态在多个设备上复制, 数据均分给各设备. 对 LLM 训练, ZeRO [19] 进一步把这些状态分片到各数据并行进程, 用 allgather 获取参数, 用 reduce-scatter 聚合梯度.

- Pipeline Parallelism (PP): Each device holds a portion of the model layers with each training batch divided into micro-batches for pipeline execution. Efficient scheduling strategies like GPipe [11], PipeDream 1F1B [12], and Zero Bubble Pipeline Parallelism (ZBPP) [33], are required to minimize “pipeline bubbles”.

• 流水线并行 (PP): 每个设备持有一部分模型层, 每个训练 batch 被切成 micro-batch 流水执行. 需要 GPipe [11], PipeDream 1F1B [12], Zero Bubble 流水线并行 (ZBPP) [33] 等调度策略来减少「流水线气泡」.

- Tensor Parallelism (TP): This involves placing a model layer on multiple GPUs that perform computations in parallel [13] [14]. It includes row-wise and column-wise parallelism, necessitating allgather and all2all for input splitting and output merging.

• 张量并行 (TP): 把一个模型层放到多张 GPU 上并行计算 [13] [14], 有按行和按列两种切法, 需要 allgather 和 all2all 来切分输入, 合并输出.

- Expert Parallelism (EP): MoE Models' different expert models are distributed on different GPUs during MoE training [15]–[17]. The gate model selects tokens for allocation during input, with corresponding tokens sent to experts model via all2all communication.

• 专家并行 (EP): MoE 训练时, 不同专家分布在不同 GPU 上 [15]-[17]. 输入时由门控模型选择 token 的去向, 再通过 all2all 把 token 发给对应专家.

- Fully Sharded Data Parallel (FSDP) is an implementation based on the ZeRO Stage 3 algorithm [19]. FSDP partitions the model's parameters, optimizer states, and gradients, distributing them across different GPUs, with each GPU retaining only $1/n$ of the total. During forward propagation, FSDP performs an allgather operation to assemble the complete parameters, which are then released after the forward pass is completed. Similarly, during backward propagation, FSDP conducts an allgather operation to obtain the complete parameters, followed by backward computation to calculate gradients. It then performs a reduce-scatter operation to synchronize gradients across all GPUs, resulting in each GPU holding $1/n$ of the reduced gradients. Finally, FSDP updates the $1/n$ parameters using each GPU's $1/n$ gradients and optimizer states. FSDP reduces GPU memory usage by main-

taining only 1/n of the parameters, gradients, and optimizer states on each GPU, enabling training of larger-scale models.

• 全分片数据并行 (FSDP) 是基于 ZeRO Stage 3 [19] 的实现. FSDP 把模型参数, 优化器状态和梯度切分到各 GPU, 每张 GPU 只保留 $1/n$. 前向时 FSDP 先 allgather 拼出完整参数, 前向结束后释放. 反向时同样先 allgather 拿到完整参数, 计算梯度后做 reduce-scatter 同步梯度, 每张 GPU 得到 $1/n$ 的规约后梯度. 最终每张 GPU 用自己的 $1/n$ 梯度和优化器状态更新 $1/n$ 参数. 由于每张 GPU 只保存 $1/n$ 的参数, 梯度和优化器状态, FSDP 降低了显存占用, 可以训练更大的模型.

There are additional strategies and algorithms to accelerate training or reduce memory usage, such as Activation Recomputation [34], [35], as well as enhanced communication and computation overlap during parallelism [36], among others.

还有其他加速训练或降低显存的策略和算法, 例如激活重计算 [34], [35], 以及并行中更充分的计算通信重叠 [36] 等.

2. Stability: The second challenge is achieving high-stability training at scale, i.e., maintaining efficient training throughout the process. Stability is vital from a production standpoint as training a big model with a trillion tokens may span several weeks. In DL training, stragglers and hardware failures are common occurrences rather than outliers. Stragglers can decelerate tasks involving hundreds of GPUs, emphasizing the importance of stability and task recovery time.

3. 稳定性: 第二个挑战是在大规模下保持整个训练过程高效. 万亿 token 的大模型训练可能持续数周, 期间掉队节点 (straggler) 和硬件故障都会发生. 一个掉队节点就能拖慢涉及数百张 GPU 的任务, 因此系统需要控制性能波动并缩短故障恢复时间.

C. HPC and AI Clusters of This Era · 当下的 HPC 与 AI 集群

1. HPC Inadequacies for AI Training: Traditional supercomputers such as TianHe-2A [37], Stampede 2 [38], and Sunway TaihuLight [39] primarily focus on double precision calculations and do not support the FP16 precision, rendering them unsuitable for DL training. Fugaku [40], despite its high performance, does not support tensor GEMM acceleration, a key component for DL workloads. Although these supercomputers may not be well-suited for DL training, their robust high-performance networks and extensive experience in large-scale cluster construction offer valuable insights and lessons for subsequent researchers.

2. 传统 HPC 不适合 AI 训练: 天河二号 A [37], Stampede 2 [38], 神威太湖之光 [39] 等传统超算主要面向双精度计算, 不支持 FP16, 不适合深度学习训练. 富岳 [40] 性能很高, 但不支持张量 GEMM 加速, 而这是深度学习负载的核心. 这些超算虽不适合深度学习训练, 其高性能网络和大规模集群建设经验对后来者仍有参考价值.

3. GPU based HPC: Supercomputers like Frontier [41], Aurora [42], Summit [43] and Perlmutter [44] utilize high-performance GPUs to tackle large-scale computations. It's worth mentioning that Perlmutter utilizes an all-flash storage system, achieving a peak bandwidth of 5TB/s. Indeed, conducting DL training on these GPU-based HPCs yields significant performance.

4. 基于 GPU 的 HPC: Frontier [41], Aurora [42], Summit [43], Perlmutter [44] 等超算用高性能 GPU 处理大规模计算. Perlmutter 采用全闪存存储系统, 峰值带宽达 5TB/s. 在这些 GPU 超算上做深度学习训练确实能获得很好的性能.

5. GPU Clusters of Large Companies: Meta, formerly Facebook, has developed its AI-HPC using a software-hardware co-design approach, with one system utilizing IB and another employing RoCE [45], [46]. ByteDance initially implemented a DL cluster with a mix of CPU and PCIe GPU [47]. However, with the advent of the LLMs era, they adopted an architecture similar to DGX, building a cluster with over 10,000 GPUs [48]. Alibaba has developed its own HPN network [49] for LLMs training using NVIDIA H800 GPUs. NVIDIA also has its own AI-HPC Eos [50], which will feature 576 DGX H100 systems totaling 4,608 H100 GPUs. While this will provide a considerable boost in computational power for AI tasks, the high cost of DGX systems raises questions about economic viability.

6. 大公司的 GPU 集群: Meta (原 Facebook) 用软硬件协同设计建设了 AI-HPC, 一套用 IB, 另一套用 RoCE [45], [46]. 字节跳动最初搭建了 CPU 与 PCIe GPU 混合的深度学习集群 [47], 进入 LLM 时代后改用类似 DGX 的架构, 建成了万卡以上的集群 [48]. 阿里巴巴为基于 NVIDIA H800 的 LLM 训练自研了 HPN 网络 [49]. NVIDIA 自己的 AI-HPC Eos [50] 将配备 576 台 DGX H100, 共 4,608 张 H100. 这会大幅提升 AI 算力, 但 DGX 系统价格高昂, 经济性值得商榷.

7. AI DSA Clusters: Custom-designed AI DSA (Domain Specific Architecture) accelerators like Google's TPU [51], utilize highly advanced optical switch reconfigurable networks. Alternatives to traditional GPU setups, like Intel Habana Gaudi [52], are also available. Teslahas introduced the Dojo [53], [54] supercomputer, which uses System on

Figure 3: Size of Model Parameter and Accelerator Memory

Wafer technology to build an entire silicon wafer as a single chip. Huawei has designed the Ascend AI DSA chip $[55]$ , $[56]$ , which remains competitive with NVIDIA, as noted by NVIDIA CEO Jensen Huang. These accelerators are tailored for efficient execution of AI workloads, offering specialized features to optimize model training and inference. However, their software ecosystems, while progressing, still require further development to match the maturity of NVIDIA's offerings.

4. AI DSA 集群: Google TPU [51] 等定制的 AI 领域专用架构 (DSA) 加速器, 采用了先进的光交换可重构网络. Intel Habana Gaudi [52] 等也是传统 GPU 之外的选择. Tesla 推出的 Dojo [53], [54] 超算采用晶圆级系统技术, 把整片硅晶圆做成一颗芯片. 华为设计了昇腾 AI DSA 芯片 [55], [56], NVIDIA CEO 黄仁勋也认为它与 NVIDIA 有竞争力. 这些加速器针对 AI 负载做了专门优化, 软件生态在进步, 但成熟度仍不及 NVIDIA.

5. Cloud Service Providers: Cloud service providers, such as Azure, offer flexible and scalable resources for AI training. Despite their convenience and easy accessibility, the costs can accumulate significantly over time. For long-term projects spanning around two years, these costs could amount to purchasing an entire dedicated cluster. Therefore, this option may not be the most economical choice for extensive AI computations.

6. 云服务商: Azure 等云服务商提供灵活可扩展的 AI 训练资源. 它们方便易得, 但成本会随时间累积, 对持续两年左右的长期项目, 云费用可能足以买下一整套专用集群. 因此对大规模 AI 计算而言, 云不一定是最经济的选择.

D. Challenges in AI Infrastructure · AI 基础设施面临的挑战

As models continue to grow larger, DL training requires thousands of GPUs. Additionally, researchers often need to train multiple models simultaneously. Therefore, a cluster with at least tens of thousands of GPUs can meet the needs of AI practitioners. In addition to increasing the scale of the cluster nodes and adding more GPUs, there's also a need to find ways to save on the overall system construction costs. These costs include but are not limited to power support, cooling, networking, storage, fault handling, disaster recovery, etc. Building a cost-effective AI-HPC system is a significant challenge. The question of how to construct a high-performance, efficient, economical, and environmentally friendly HPC to meet AI training requirements has become a hot topic. Some works, such as [57] analyzes the construction of HPC, discussed the interconnection of heterogeneous clusters, cooling systems. AI applications such as Peng Cheng Cloud Brain II [58] discussed their strategies for building and improving AI computing power and cluster communication efficiency, which helped them achieve first place in IO500 and AIPerf rankings.

模型越来越大, 深度学习训练需要数千张 GPU, 研究者还常常要同时训练多个模型, 因此至少要数万张 GPU 的集群才能满足 AI 从业者的需求. 除了扩大节点规模, 增加 GPU, 还要设法降低整体建设成本, 包括但不限于供电, 制冷, 网络, 存储, 故障处理, 容灾等. 建设高性价比的 AI-HPC 系统是一项重大挑战, 如何建设高性能, 高效, 经济, 环保的 HPC 来满足 AI 训练需求已成为热点. [57] 等工作分析了 HPC 建设, 讨论了异构集群互连和制冷系统. 鹏城云脑 II [58] 等 AI 应用讨论了提升 AI 算力和集群通信效率的策略, 帮助它们在 IO500 和 AIPerf 榜单上取得第一.

Drawing from our extensive experience in Deep Learning spanning over the past decade, we have conducted considerable exploration in terms of cost-effectiveness. This work primarily discusses our practices for achieving cost-effectiveness and high-performance across different models and stages.

基于过去十余年深度学习的经验, 我们在性价比方面做了大量探索. 本文主要讨论我们在不同模型, 不同阶段实现高性价比和高性能的实践.

Figure 4: In-node Architecture: 8 PCIe GPUs and 1 InfiniBand (IB) NIC are directly connected to the CPU. Note that GPU5/6 share the same PCIe root port, while IB occupies one independently.

III. FIRE-FLYER 2: OUR APPROACH FOR DEEP LEARNING AND EARLY LLM TRAINING · Fire-Flyer 2: 面向深度学习与早期 LLM 训练的方案

As mentioned in the Background Section, LLMs generally require significant memory resources. In contrast, many other models necessitate considerably less memory, as illustrated in Figure 3. Popular models like ResNet [22], Mask-RCNN [59], BERT [60], MAE [61], among others, all have a parameter volume less than 1B, signifying relatively low memory requirements. Therefore, when designing a cluster primarily for deep learning model training, and with insights gleaned from our Fire-Flyer 1 experiments, we deemed it prudent to incorporate PCIe A100 GPUs, which proved to be sufficient during its construction in 2021.

如背景部分所述, LLM 通常需要大量内存. 相比之下, 很多其他模型所需内存要少得多, 见 Figure 3. ResNet [22], Mask-RCNN [59], BERT [60], MAE [61] 等常用模型参数量都小于 1B, 内存需求相对较低. 因此在设计一个主要用于深度学习训练的集群时, 结合 Fire-Flyer 1 的经验, 我们认为采用 PCIe A100 是稳妥的, 它在 2021 年建设时也确实够用.

A. Fire-Flyer 2: PCIe A100 GPU Architecture · Fire-Flyer 2: PCIe A100 GPU 架构

In our training workloads, the bandwidth requirements for both storage IOs and computation communication across 8 NVIDIA PCIe A100 GPUs can be met by a single 200Gbps NVIDIA Mellanox ConnectX-6 (CX6) InfiniBand (IB) NIC. We employed the following computation node architecture, as shown in Figure 4:

在我们的训练负载中, 8 张 NVIDIA PCIe A100 的存储 IO 和计算通信带宽需求, 用一张 200Gbps 的 NVIDIA Mellanox ConnectX-6 (CX6) InfiniBand (IB) 网卡就能满足. 我们采用如下计算节点架构, 见 Figure 4:

- 8 NVIDIA A100 PCIe GPUs and 1 Mellanox CX6 200Gbps IB NIC: directly connect to the CPU, without using a PCIe switch

• 8 张 NVIDIA A100 PCIe GPU 和 1 张 Mellanox CX6 200Gbps IB 网卡: 直接连到 CPU, 不使用 PCIe switch

- IB NIC occupies a separate PCIe root complex, thus avoiding performance interference with the GPU.

• IB 网卡独占一个 PCIe root complex, 避免与 GPU 互相干扰性能.

- Reserved the possibility of NVLink Bridge addition in design: As expected, when the LLM era arrived, we indeed added an NVLink Bridge between PCIe cards.

• 设计时预留了加装 NVLink Bridge 的可能: 不出所料, LLM 时代到来后, 我们确实在 PCIe 卡之间加装了 NVLink Bridge.

Table I shows our arch details and compared with NVIDIA standard DGX-A100 server.

Table I 给出了我们的架构细节, 并与 NVIDIA 标准 DGX-A100 服务器对比.

B. Network Topology: Two-Layer Fat-Tree with Storage and Computation Integrated · 网络拓扑: 存算一体的两层 Fat-Tree

We selected the Fat-Tree [9] topology as our primary network architecture due to its exceptionally high bisection bandwidth, making it the preferred choice for AI-HPC and high-throughput storage environments. Although the Dragonfly topology [62], [63] also offers comparable cost-effectiveness and performance, its lack of sufficient bisection bandwidth makes it unsuitable for our integrated storage and computation network design. At the time of implementation, various RoCE (RDMA over Converged Ethernet) [64] technologies were not as mature as they are today, so we opted for InfiniBand (IB) as our network solution. Mellanox QM8700 InfiniBand Switch, offering 40 ports at 200 Gbps, was utilized. Our cluster, consisting of 10,000 A100 GPUs, includes approximately 1,250 GPU compute nodes and nearly 200 storage servers, although a Two-Layer Fat-Tree can accommodate up to 800 nodes (configured with 20 spine switches and 40 leaf switches).

我们选择 Fat-Tree [9] 作为主网络架构, 因为它的二分带宽极高, 是 AI-HPC 和高吞吐存储环境的首选. Dragonfly 拓扑 [62], [63] 的性价比和性能也相近, 但二分带宽不足, 不适合我们存算一体的网络设计. 建设时各种 RoCE [64] 技术还不如今天成熟, 所以我们选了 InfiniBand (IB). 交换机用 Mellanox QM8700, 40 个 200Gbps 端口. 集群有 10,000 张 A100, 约 1,250 个 GPU 计算节点和近 200 台存储服务器, 而一个两层 Fat-Tree (20 台 spine, 40 台 leaf) 最多只能接 800 个节点.

To reduce costs, , we opted for a two-zone network configuration instead of a three-layer Fat-Tree solution, as shown in Figure 5. Each zone consists of an 800-port Fat-Tree connected to approximately 600 GPU compute nodes. Each storage server equipped with two IB NICs, respectively connected to different zones, hence all GPU compute nodes could share a set of storage services. Additionally, the two zones are interconnected with a limited number of links. Our HAI Platform scheduling strategy ensured that cross-zone computing tasks were limited to one at most. Whether using NCCL [10] or our in-house developed communication library HFReduce, it can be run across zones by using a double binary tree algorithm [65]. Our scheduler ensures that in this topology, only one pair of nodes communicates across zones. Consequently,

Figure 5: Network Topology: Two complete Two-Layer Fat-Tree connected together.

even tasks requiring all nodes can efficiently run on the entire Fire-Flyer 2 AI-HPC.

为降低成本, 我们没有采用三层 Fat-Tree, 而是用两区网络配置, 见 Figure 5. 每个区是一个 800 端口的 Fat-Tree, 接约 600 个 GPU 计算节点. 每台存储服务器配两张 IB 网卡, 分别接入两个区, 所以全部 GPU 计算节点可以共享同一套存储服务. 两个区之间只有少量链路互连. HAI Platform 的调度策略保证同一时间最多只有一个跨区计算任务. 无论 NCCL [10] 还是自研的 HFReduce, 都可以用双二叉树算法 [65] 跨区运行, 调度器保证在这种拓扑下只有一对节点跨区通信. 因此, 即使是需要全部节点的任务, 也能在整个 Fire-Flyer 2 AI-HPC 上高效运行.

C. Cost Performance of Our Architecture · 本架构的性价比

Compared to the NVIDIA DGX-A100 [8] architecture, our approach using PCIe A100 achieves approximately 83% of the performance in TF32 and FP16 General Matrix Multiply (GEMM) benchmarks. However, it offers substantial reductions in both costs and energy usage, achieving 60% of the GPU cost and energy consumption, as detailed in Table II.

与 NVIDIA DGX-A100 [8] 架构相比, 我们基于 PCIe A100 的方案在 TF32 和 FP16 通用矩阵乘 (GEMM) 测试中达到约 83% 的性能, 同时成本和能耗大幅下降, GPU 成本和能耗都只有 60%, 见 Table II.

Contrasting with the DGX-A100 cluster, which necessitates a Three-Layer Fat-Tree encompassing 10,000 access points and involving 320 core switches, 500 spine switches and 500 leaf switches, amounting to 1,320 switches in total (as shown in Table III), our architecture only requires 122 switches. This arrangement is significantly more cost-efficient. Even when compared to a similarly sized three-layer Fat-Tree network with 1,600 access points, which includes 40 core switches and 160 spine and leaf switches (totaling 200 switches), our design facilitates a saving of 40% in networking costs.

DGX-A100 集群需要一个三层 Fat-Tree, 接入 10,000 个端点, 包含 320 台 core, 500 台 spine 和 500 台 leaf 交换机, 共 1,320 台 (见 Table III), 我们的架构只需要 122 台, 成本低得多. 即使与规模相当的三层 Fat-Tree (1,600 个接入点, 40 台 core 加 160 台 spine 和 leaf, 共 200 台) 相比, 我们的设计也节省了 40% 的网络成本.

Furthermore, by utilizing an 800-Ports Frame Switch, we have further reduced the cost of optical modules and cables. While there is a performance gap due to the inherent differences between PCIe card specifications and SXM, we generally achieved 80% of the DGX-A100 performance at merely 60% of the cost. Additionally, we managed to trim energy consumption by 40%, thereby reducing $CO_{2}$ emissions. In terms of cost-performance, we regard this approach as both effective and successful.

此外, 使用 800 端口的框式交换机进一步降低了光模块和线缆成本. PCIe 卡与 SXM 规格的固有差异带来一定性能差距, 但总体上我们以 60% 的成本拿到了 DGX-A100 80% 的性能, 能耗降低 40%, 也减少了 $CO_2$ 排放. 从性价比看, 我们认为这一方案是有效且成功的.

IV. HFREDUCE: HARDWARE SOFTWARE CO-DESIGN IN NETWORK · HFReduce: 网络层面的软硬件协同设计

In large-scale deep learning training, allreduce is essential for aggregating gradients across GPUs. To optimize communication among PCIe GPUs in our architecture, we developed HFReduce, a library specifically designed for efficient allreduce operations. The core strategy of HFReduce, illustrated in Figure 6, involves performing intra-node reduction first, followed by inter-node allreduce of the reduced data from the 8

Figure 6: HFReduce Schematic: 1) do intra-node reduce, 2) do inter-node allreduce by CPU, 3) finally transfer reduced data to GPU.

GPUs within each node. This inter-node allreduce leverages a Double Binary Tree Algorithm [65], akin to NCCL, and is pipelined by dividing data into chunks for transfer via Remote Direct Memory Access (RDMA), ensuring high performance. HFReduce is versatile and can be applied to any scenario requiring allreduce, as well as general reduce and broadcast operations.

大规模深度学习训练中, allreduce 是跨 GPU 聚合梯度的必需操作. 为优化本架构中 PCIe GPU 之间的通信, 我们开发了专门做高效 allreduce 的库 HFReduce. 其核心策略见 Figure 6: 先做节点内规约, 再对每个节点 8 张 GPU 规约后的数据做节点间 allreduce. 节点间 allreduce 采用与 NCCL 类似的双二叉树算法 [65], 并把数据切块, 通过 RDMA 流水传输, 保证高性能. HFReduce 适用于任何需要 allreduce 的场景, 也可用于一般的 reduce 和 broadcast 操作.

A. HFReduce Algorithm Steps · HFReduce 算法步骤

Intra-node reduction, as shown in the Algorithm 1:

节点内规约, 见 Algorithm 1:

1. When the gradients data on the GPUs require allreduce, HFReduce asynchronously transfers these data to the CPU memory. This Device-To-Host (D2H) Transfer can utilize GDRCopy [66] for small data and Mem-CpyAsync for larger data.

2. 当 GPU 上的梯度需要 allreduce 时, HFReduce 把数据异步传到 CPU 内存. 这一步是设备到主机 (D2H) 传输, 小数据可用 GDRCopy [66], 大数据用 MemCpyAsync.

3. Upon the arrival of the gradients in memory, perform reduction add operation using CPU vector instructions.

4. 梯度到达内存后, 用 CPU 向量指令做规约加法.

Inter-node reduction, as shown in the Algorithm 2:

节点间规约, 见 Algorithm 2:

1. Use the Double Binary Tree Algorithm [65] for inter-node allreduce, facilitating transfers between nodes using RDMA verbs implementation.

2. 用双二叉树算法 [65] 做节点间 allreduce, 节点间传输基于 RDMA verbs 实现.

3. Finally, the CPU returns reduced gradients to the GPU via PCIe (Host-To-Device Phase).

4. 最终 CPU 通过 PCIe 把规约后的梯度传回 GPU (主机到设备阶段).

The final Host-To-Device (H2D) Transfer can be optimized by utilizing GDRCopy to write data to four GPUs within the same NUMA node, effectively reducing reads from host memory by threefold compared to MemCpyAsync. This efficiency is achieved because GDRCopy can read data from host memory and temporarily cache it in the CPU caches, allowing data to be written to the four GPUs without additional reads from host memory.

末尾的主机到设备 (H2D) 传输可以用 GDRCopy 优化: 把数据写到同一 NUMA 节点内的 4 张 GPU, 主机内存读取量降为 MemCpyAsync 的约三分之一. 原因是 GDRCopy 从主机内存读出数据后会暂存在 CPU cache 里, 写 4 张 GPU 时不必再次读主机内存.

B. Advantages of HFReduce over NCCL · HFReduce 相对 NCCL 的优势

1. Reduced PCIe Bandwidth Consumption: Let $n$ be the total number of GPUs involved in the communication. In NCCL's ring topology, each unit of data needs to go through $2n - 1$ transmissions, each consuming one unit of inbound

Algorithm 1: Intra Node Reduce.
Data: Dg: data need to allreduce
Result: Dc: data reduced in this node
Split Dg by Chunk_Size
for Dg-i in Splited-Dg do
    // Transfer GPU Memory Data to CPU memory
    Dc_i = MemCopyAsync Dg_i to CPU Mmeory
end
for i in Splited_Count do
    while every GPU's Dg_i → Dc_i finished do
        // Wait for chunk-i transfer
            finished in this node
        for j in GPU_Count do
            // do intra-node reduce
            Dc_i += GPU-j's Dc_i
        end
        // do inter-node reduce
        Do_Internode_reduce(Dc_i)
    end
end

bandwidth of one GPU and one unit of outbound bandwidth of another GPU. This means for a single unit of data, it consumes $\frac{2n-1}{n}$ unit of PCIe bidirectional bandwidth. In contrast, for each unit of data, HFReduce only requires one D2H and one H2D data transfer, only one unit of PCIe bidirectional bandwidth is consumed. In our machine architecture, the performance of NCCL is mainly limited by PCIe bandwidth. Therefore, HFReduce can achieve better performance than NCCL.

1. 减少 PCIe 带宽消耗: 设参与通信的 GPU 总数为 $n$. 在 NCCL 的 ring 拓扑中, 每单位数据要经过 $2n-1$ 次传输, 每次消耗一张 GPU 一单位入向带宽和另一张 GPU 一单位出向带宽. 也就是说, 每单位数据消耗 $\frac{2n-1}{n}$ 单位的 PCIe 双向带宽. HFReduce 对每单位数据只需一次 D2H 和一次 H2D, 只消耗一单位 PCIe 双向带宽. 在我们的机器架构上, NCCL 的性能主要受 PCIe 带宽限制, 因此 HFReduce 能比 NCCL 更快.

拆开: $\frac{2n-1}{n}$ 是怎么来的, 和常见的 ring allreduce 计数对得上吗?

答: 按 IV-B1 的口径, $2n-1$ 次传输由 $n$ 张 GPU 分摊, 每张 GPU 摊到 $\frac{2n-1}{n}$ 单位, $n$ 大时趋近 2. 通常的 ring allreduce 计数是 reduce-scatter 与 allgather 各走 $n-1$ 步, 每张 GPU 收发各 $\frac{2(n-1)}{n}$ 单位, 同样趋近 2. HFReduce 一次 D2H 加一次 H2D 合计 1 单位, 两者相差约 2 倍, 与 IV-B 末尾给出的 HFReduce 6.3-8.1GB/s 对 NCCL 1.6-4.8GB/s 的量级相符. 原文的 $2n-1$ 比标准计数多出约 1 次, 不影响「约 2 倍」的结论.

2. No GPU Kernel Overhead: HFReduce utilizes the GPU's Copy Engine (CE) for PCIe asynchronous transfers. In contrast, NCCL's allreduce operation requires GPU kernel execution, which can affect other computational kernels on the GPU. HFReduce achieves complete asynchrony with no overhead.

3. 没有 GPU kernel 开销: HFReduce 用 GPU 的 Copy Engine (CE) 做 PCIe 异步传输, NCCL 的 allreduce 则需要执行 GPU kernel, 会影响 GPU 上其他计算 kernel. HFReduce 做到了完全异步, 没有额外开销.

As demonstrated in Figure 7a, HFReduce can reach a inter-node bandwidths of 6.3-8.1GB/s when performing allreduce with a data size of 186 MiB on the Fire-Flyer 2 AI-HPC, while NCCL's inter-node bandwidth is only 1.6-4.8GB/s.

如 Figure 7a 所示, 在 Fire-Flyer 2 AI-HPC 上对 186 MiB 数据做 allreduce 时, HFReduce 的节点间带宽可达 6.3-8.1GB/s, NCCL 只有 1.6-4.8GB/s.

C. Performance Improvements: HFReduce with NVLink · 性能改进: 带 NVLink 的 HFReduce

By installing the NVLink Bridge for PCIe A100 GPUs, efficient communication is enabled between paired GPUs via the 600 GB/s NVLink. To alleviate the memory bound issue of the original HFReduce, we implemented another allreduce pattern, termed HFReduce with NVLink. The core concept involves initially performing a reduction operation among GPUs interconnected by NVLink before passing the gradient to the CPU. Subsequently, when the CPU returns the result, it splits the result data and returns them to the paired GPUs connected by NVLink respectively, then performs allgather via NVLink. As illustrated in Figure 7b, HFReduce with NVLink achieves inter-node bandwidths exceeding 10 GB/s.

给 PCIe A100 装上 NVLink Bridge 后, 成对的 GPU 之间可以通过 600GB/s 的 NVLink 高效通信. 为缓解原版 HFReduce 的内存带宽瓶颈, 我们实现了另一种 allreduce 模式, 称为 HFReduce with NVLink. 核心做法是先在 NVLink 相连的 GPU 之间做规约, 再把梯度交给 CPU. CPU 返回结果时, 把结果切开分别还给 NVLink 相连的两张 GPU, 两者再通过 NVLink 做 allgather. 如 Figure 7b 所示, HFReduce with NVLink 的节点间带宽超过 10GB/s.

(a) HFRudce and NCCL Allreduce Speed.

(b) HFReduce with NVLink (Cross Fat-Tree Zone).

Algorithm 2: Inter Node Reduce. Data: DL:local node reduced data by Algorithm 1 Data: DR: received other node reduced data Result: Dg: reduced data transfer to GPU // Pass 1: reduce data,individual thread for i in Chunk_Size do receive data $DR_i$