本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

Inside Prometheus: Meta’s Journey to a Gigawatt-Scale Supercluster

By Kevin Lee, Yashar Bayani, Ahmed Sharif, Yee Jiun Song With the recent release of the Muse family of foundational models, we want to take you inside the infrastructure that has made it all possible. The first phase of Prometheus, our gigawatt-scale supercluster, is already complete and we’re working toward scaling it to comprise ~1 million GPUs. Prometheus also features technologies like closed-loop liquid cooling to help it operate efficiently.

随着 Muse 系列基础模型的发布，我们希望带您深入了解使其成为可能的底层基础设施。Prometheus 是我们吉瓦级（gigawatt-scale）超级集群，其第一阶段已经完成，我们正致力于将其扩展至包含约 100 万个 GPU。Prometheus 还采用了闭环液冷等技术，以帮助其高效运行。

Right now, in New Albany, Ohio, Meta is undertaking its most ambitious AI infrastructure project to date. Prometheus, our gigawatt-scale supercluster, is an advancement in our infrastructure that is training the next generation of foundational models that will bring new AI experiences to billions of people. Two years ago, New Albany was a conventional data center site running web traffic. With advancements in hardware, networking and building design, we reworked this site to be home to training programs for frontier models.

目前，Meta 正在俄亥俄州新奥尔巴尼（New Albany）开展迄今为止最雄心勃勃的 AI 基础设施项目。Prometheus 作为我们的吉瓦级超级集群，是我们基础设施的一次飞跃，它正在训练下一代基础模型，旨在为数十亿人带来全新的 AI 体验。两年前，新奥尔巴尼还是一个处理网络流量的传统数据中心。随着硬件、网络和建筑设计的进步，我们对该站点进行了改造，使其成为前沿模型训练项目的基地。

We’re building the supercluster in phases. The first phase has already been completed and was used to train our Muse foundational models, including the Muse Wonder models, our new frontier models that deliver state-of-the-art performance on personal and professional agentic tasks. Large training jobs require everything to work optimally to sustain their performance; an issue on a single host can slow down or stop the entire job, lowering its ‘goodput’.

我们正分阶段构建该超级集群。第一阶段已经完成，并被用于训练我们的 Muse 基础模型，包括 Muse Wonder 模型——这是我们全新的前沿模型，在个人和专业智能体任务中提供了最先进的性能。大型训练任务需要一切环节协同工作以维持性能；单个主机上的问题就可能导致整个任务减速或停止，从而降低其“有效吞吐量”（goodput）。

By focusing on improving our systems’ stability and reliability we were able to achieve more than 85% goodput for pre-training runs and more than 90% goodput for reinforcement learning (RL) runs of Muse Wonder. During these training runs, we were able to achieve an overall wall-clock mean time between interruptions (MTBI) of ~47k GPU hours. Since then, we have landed further changes across our fleet including firmware fixes to increase MTBI to more than 54K GPU hours, and we are constantly improving it.

通过专注于提高系统的稳定性和可靠性，我们在 Muse Wonder 的预训练运行中实现了超过 85% 的有效吞吐量，在强化学习（RL）运行中实现了超过 90% 的有效吞吐量。在这些训练过程中，我们实现了约 4.7 万 GPU 小时的平均中断间隔时间（MTBI）。此后，我们在整个集群中实施了进一步的改进，包括固件修复，将 MTBI 提高到了超过 5.4 万 GPU 小时，并且我们仍在不断优化。

Now, we’re continuing to scale Prometheus with our next frontier models from Meta Superintelligence Labs in mind. The Prometheus supercluster is housed in Meta’s New Albany, Ohio, data center. The Supercluster: Unlocking New AI Capabilities Today, Prometheus comprises an array of hundreds of thousands of GPUs across multiple interconnected buildings, including traditional and AI-optimized data center buildings and rapid deployment structures, all tightly interconnected by miles of fiber and functioning as a single machine.

现在，我们正怀揣着来自 Meta 超级智能实验室（Meta Superintelligence Labs）的下一代前沿模型，继续扩展 Prometheus。Prometheus 超级集群位于 Meta 位于俄亥俄州新奥尔巴尼的数据中心内。超级集群：解锁新的 AI 能力。如今，Prometheus 由分布在多个互联建筑中的数十万个 GPU 组成，其中包括传统数据中心、针对 AI 优化的数据中心以及快速部署结构，所有这些都通过数英里的光纤紧密连接，并作为一个单一的机器运行。

This capacity allows us to pursue larger models and longer context windows than were previously practical. At full scale, Prometheus will comprise roughly 1 million distributed GPUs, all functioning in lockstep. Multiple Generations of AI Hardware Prometheus currently deploys two NVIDIA GPU platforms, with a third (based on NVIDIA’s Vera Rubin NVL72) arriving soon. Each represents a generational leap in performance and efficiency, leveraging our NVL72 rack architecture where 72 GPUs act as a single accelerator within the same network fabric.

这种能力使我们能够追求比以往更大型的模型和更长的上下文窗口。在完全规模化后，Prometheus 将包含约 100 万个分布式 GPU，所有这些 GPU 都将同步运行。多代 AI 硬件：Prometheus 目前部署了两个 NVIDIA GPU 平台，第三个平台（基于 NVIDIA 的 Vera Rubin NVL72）也即将到来。每一个平台都代表了性能和效率的代际飞跃，利用了我们的 NVL72 机架架构，其中 72 个 GPU 在同一个网络结构内作为一个单一的加速器运行。

Catalina: NVIDIA GB200 Our first platform, Catalina, leverages the NVIDIA GB200 NVL72. We deploy the Ariel variant, a two-rack configuration in which 36 GPUs per rack are interconnected via NVLink to form a unified 72-GPU domain. Catalina provides improvements in raw compute and memory capacity compared to Grand Teton H100. It leverages our Open Rack V3 (ORv3) high-power rack (HPR) structure, an open hardware design contributed to the Open Compute Project (OCP).

Catalina：NVIDIA GB200。我们的第一个平台 Catalina 利用了 NVIDIA GB200 NVL72。我们部署了 Ariel 变体，这是一种双机架配置，每个机架 36 个 GPU 通过 NVLink 互联，形成一个统一的 72-GPU 域。与 Grand Teton H100 相比，Catalina 在原始计算能力和内存容量上都有所提升。它利用了我们的 Open Rack V3 (ORv3) 高功率机架（HPR）结构，这是一种贡献给开放计算项目（OCP）的开放硬件设计。

The Ariel variant of GB200 NVL72 pairs each NVIDIA Blackwell GPU with an NVIDIA Grace CPU and 2x NVIDIA ConnectX-7 400G NICs at a 1:1 CPU-to-GPU ratio, letting us deploy 800G of I/O per GPU earlier than the rest of the industry. We chose Ariel because the GB200 NVL72 scaled up nearly every other dimension relative to the NVIDIA H100: FLOPS, HBM bandwidth, scale-up bandwidth, and scale-up domain size, while scale-out bandwidth per GPU would have stayed at 400G.

GB200 NVL72 的 Ariel 变体将每个 NVIDIA Blackwell GPU 与一个 NVIDIA Grace CPU 和两个 NVIDIA ConnectX-7 400G 网卡配对，CPU 与 GPU 的比例为 1:1，使我们能够比行业其他公司更早地部署每个 GPU 800G 的 I/O。我们选择 Ariel 是因为 GB200 NVL72 在几乎所有维度上都比 NVIDIA H100 有所扩展：FLOPS、HBM 带宽、纵向扩展（scale-up）带宽和纵向扩展域大小，而每个 GPU 的横向扩展（scale-out）带宽本应保持在 400G。

We expected that imbalance to cost performance, so we opted for a configuration that delivers 800G of scale-out per GPU. This also gave us more uniformity of spec as the region expanded over time and 800G scale-out became available. The Catalina GB200 NVL72 features: 72 NVIDIA Blackwell GPUs with 13.4 TB of HBM3e (192 GB per GPU). 720 peak PFLOPS of FP8 training compute (1,440 PFLOPS at FP4). 130 TB/s peak NVLink bandwidth within the domain. 576 TB/s peak aggregate memory bandwidth. ~120 kW total system power across the two-rack pair.

我们预计这种不平衡会影响性能，因此我们选择了提供每个 GPU 800G 横向扩展带宽的配置。随着时间的推移，当 800G 横向扩展变得可用时，这也为我们在区域扩展时提供了更统一的规格。Catalina GB200 NVL72 的特点包括：72 个 NVIDIA Blackwell GPU，配备 13.4 TB HBM3e（每个 GPU 192 GB）；720 PFLOPS 的 FP8 训练计算峰值（FP4 下为 1440 PFLOPS）；域内 130 TB/s 的 NVLink 带宽峰值；576 TB/s 的总内存带宽峰值；双机架总系统功耗约为 120 kW。

Clemente: NVIDIA GB300 NVL72 Our second platform, Clemente, moves to the NVIDIA GB300 NVL72 and improves on Catalina at both the GPU and the platform level. GB300 raises HBM capacity per GPU by 1.5x, from 192 GB to 288 GB, and dense FP4 throughput by a similar factor, while roughly doubling attention (softmax) acceleration. At the platform level, scale-out is no longer the constraint it was on the GB200 NVL72: NVIDIA ConnectX-8 SuperNICs deliver 800G per GPU natively, so the capability we deployed early on Ariel becomes the baseline.

Clemente：NVIDIA GB300 NVL72。我们的第二个平台 Clemente 转向了 NVIDIA GB300 NVL72，并在 GPU 和平台层面都对 Catalina 进行了改进。GB300 将每个 GPU 的 HBM 容量提高了 1.5 倍，从 192 GB 增加到 288 GB，密集 FP4 吞吐量也提高了类似的倍数，同时将注意力（softmax）加速能力大约翻了一番。在平台层面，横向扩展不再像 GB200 NVL72 那样成为瓶颈：NVIDIA ConnectX-8 SuperNIC 原生提供每个 GPU 800G 的带宽，因此我们在 Ariel 上早期部署的能力成为了基准。

That, together with a 2:1 GPU-to-CPU ratio, allows Clemente to pack the full 72-GPU domain into a single rack, streamlining our physical deployment. Clemente can be deployed with our modular Air-Assisted Liquid Cooling (AALC) or use Facility Liquid Cooling (FLC) in our newly designed data centers. Serviceability also improves substantially, with reduced cabling. The Clemente GB300 NVL72 features: 72 NVIDIA Blackwell Ultra GPUs with 20.7 TB of HBM3e (288 GB per GPU — 50% more than the GB200). Over 1 exaFLOP of peak FP4 compute. 130 TB/s peak NVLink bandwidth (1.8 TB/s bidirectional per GPU). ~135 kW rack power. 5-10% performance improvement over the GB200 NVL72.

这一点，加上 2:1 的 GPU 与 CPU 比例，使得 Clemente 能够将完整的 72-GPU 域封装在单个机架中，从而简化了我们的物理部署。Clemente 可以通过我们的模块化空气辅助液冷（AALC）进行部署，或者在我们新设计的数据中心中使用设施液冷（FLC）。可维护性也得到了显著提高，布线需求减少。Clemente GB300 NVL72 的特点包括：72 个 NVIDIA Blackwell Ultra GPU，配备 20.7 TB HBM3e（每个 GPU 288 GB，比 GB200 多 50%）；超过 1 exaFLOP 的 FP4 计算峰值；130 TB/s 的 NVLink 带宽峰值（每个 GPU 双向 1.8 TB/s）；约 135 kW 的机架功耗；比 GB200 NVL72 性能提升 5-10%。