MetaRoCE: A New RDMA Transport Built for AI-Scale Ethernet

MetaRoCE：专为 AI 规模以太网打造的全新 RDMA 传输协议

By Arvind Srinivasan, Neil Spring, Omar Baldonado, Rajiv Krishnamurthy 作者：Arvind Srinivasan, Neil Spring, Omar Baldonado, Rajiv Krishnamurthy

Training and serving frontier AI models depends on fast, reliable networks that move data between GPUs without wasting compute cycles. To meet this challenge at scale, Meta designed MetaRoCE – a clean-sheet RDMA transport protocol purpose-built for AI workloads on commodity Ethernet. We’re releasing the MetaRoCE specification, a reference software implementation and a compliance test suite through the Open Compute Project (OCP) to enable the broader industry to adopt, implement, and build on it. 训练和部署前沿 AI 模型依赖于快速、可靠的网络，以便在 GPU 之间传输数据而不浪费计算周期。为了应对这一大规模挑战，Meta 设计了 MetaRoCE——一种专为通用以太网上的 AI 工作负载而全新构建的 RDMA 传输协议。我们正通过开放计算项目 (OCP) 发布 MetaRoCE 规范、参考软件实现以及合规性测试套件，旨在使整个行业能够采用、实现并在此基础上进行构建。

At Meta, we’ve been a strong driver behind the industry’s growing consensus that Ethernet should be the fabric of choice for AI infrastructure. We’ve already shown that RoCE can power distributed AI training at scale. Now, we’re building on that work with MetaRoCE, a protocol designed from the ground up for Ethernet at million-GPU scale. 在 Meta，我们一直是推动行业共识的核心力量，即以太网应成为 AI 基础设施的首选网络架构。我们已经证明 RoCE 可以支持大规模分布式 AI 训练。现在，我们在此基础上推出了 MetaRoCE，这是一种从底层开始为百万级 GPU 规模的以太网而设计的协议。

We’ve scaled up clusters of hundreds of thousands of GPUs, spread over multiple data centers and regions. Whether these clusters are training the next frontier model or serving inference at global scale, the network is in the critical path. Collective operations like all-reduce and all-to-all synchronize thousands of accelerators during training, and the slowest transfer sets the pace for the entire job. In inference, low-latency communication between distributed model shards directly impacts response times for hundreds of millions of users. Even small amounts of network friction directly strand significant compute capacity. 我们已经扩展了拥有数十万个 GPU 的集群，这些集群分布在多个数据中心和区域。无论这些集群是在训练下一个前沿模型，还是在全球范围内提供推理服务，网络都处于关键路径上。诸如 all-reduce 和 all-to-all 之类的集合通信操作在训练期间同步数千个加速器，而最慢的传输决定了整个作业的进度。在推理中，分布式模型分片之间的低延迟通信直接影响数亿用户的响应时间。即使是少量的网络摩擦也会直接导致巨大的计算能力闲置。

Standard RoCE expects the network to deliver every frame in order, leveraging PFC and discouraging the packet spraying that provides performance in multiplane and large scale networks. MetaRoCE is built to provide high throughput, low tail latency, and operational simplicity as the network grows in the number of accelerators and the distances between them. 标准 RoCE 要求网络按顺序交付每一帧，利用 PFC（基于优先级的流量控制）并抑制在多平面和大规模网络中能提供高性能的数据包喷洒（packet spraying）技术。MetaRoCE 的构建旨在随着加速器数量和它们之间距离的增加，提供高吞吐量、低尾延迟和操作简便性。

How MetaRoCE Works

MetaRoCE 的工作原理

MetaRoCE’s core insight is simple: The fabric sees packets, but the NIC sees intent. Traditional architectures centralize intelligence in the fabric, relying on switches to enforce losslessness and maintain order. By moving intelligence to the endpoint, MetaRoCE decomposes the network into many fine-grained logical paths, each with its own real-time telemetry – per-path RTT, ECN state, and utilization. This visibility unlocks capabilities that are difficult to achieve with traditional RDMA. MetaRoCE 的核心洞察很简单：网络架构看到的是数据包，而网卡 (NIC) 看到的是意图。传统架构将智能集中在网络架构中，依赖交换机来强制实现无损并维持顺序。通过将智能转移到端点，MetaRoCE 将网络分解为许多细粒度的逻辑路径，每条路径都有自己的实时遥测数据——包括每条路径的 RTT（往返时间）、ECN（显式拥塞通知）状态和利用率。这种可见性解锁了传统 RDMA 难以实现的功能。

Native Out-of-Order Delivery

原生乱序交付

MetaRoCE sprays packets across many paths, so they arrive out of order by design. The transport treats out-of-order arrival as the normal case. Every packet carries its own destination, so data is written straight to its final memory location as it lands, with no reorder buffer and no head-of-line blocking. Writes carry their destination in every packet. Sends carry the match to a posted receive buffer, so a Send lands correctly even when the messages ahead of it have not arrived, and without a round trip to learn where the data goes. Collective libraries can use two-sided messaging where it suits them rather than reducing everything to Write. MetaRoCE 将数据包喷洒到多条路径上，因此它们在设计上就是乱序到达的。该传输协议将乱序到达视为常态。每个数据包都携带自己的目的地，因此数据在到达时直接写入其最终内存位置，无需重排序缓冲区，也不会出现队头阻塞。写入操作在每个数据包中都携带其目的地。发送操作携带与已发布接收缓冲区的匹配信息，因此即使在它之前的消息尚未到达时，发送操作也能正确落地，且无需往返确认即可获知数据去向。集合通信库可以在适合的地方使用双向消息传递，而不是将所有操作都简化为写入。

Native Multipathing

原生多路径

MetaRoCE gives each connection first-class paths and sprays across them packet by packet. Each path carries a distinct UDP source port as its ECMP entropy, which the NIC can change at any time to move traffic off a bad route. On multiplane fabrics, plane selection falls entirely to the NIC, and the fabric is used only as well as the NIC sprays. Because each path keeps its own window and round trip estimate, the transport can tell congestion from failure and rebalance explicitly, so a hot or broken link slows one path instead of stalling the connection. MetaRoCE 为每个连接提供了一流的路径，并逐包进行喷洒。每条路径都携带一个独特的 UDP 源端口作为其 ECMP（等价多路径）熵，网卡可以随时更改该端口，从而将流量从故障路由中移出。在多平面网络架构中，平面选择完全由网卡决定，网络架构的性能仅取决于网卡的喷洒效果。由于每条路径都保持自己的窗口和往返时间估计，传输协议可以区分拥塞和故障并进行显式重新平衡，因此热点或损坏的链路只会减慢一条路径的速度，而不会导致整个连接停滞。

Loss Tolerance by Design

设计上的容损性

MetaRoCE treats the Ethernet fabric as lossy and does not ask it to be otherwise – no PFC, no pause frames. Because each path carries its own ordered sequence, a gap in its 256-bit selective acknowledgment bitvector is evidence of loss rather than of reordering. In other protocols a SACK mostly avoids resending data that already arrived; here it triggers retransmission of exactly the missing packet, on the path that lost it, the moment the gap appears. MetaRoCE 将以太网架构视为有损的，并不要求其改变——无需 PFC，无需暂停帧。由于每条路径都携带自己的有序序列，其 256 位选择性确认 (SACK) 位向量中的间隙是丢失的证据，而不是重排序的证据。在其他协议中，SACK 主要用于避免重发已到达的数据；而在 MetaRoCE 中，一旦出现间隙，它会立即在丢失数据的路径上触发对缺失数据包的精确重传。

Congestion Control From Both Sides

双向拥塞控制

MetaRoCE combines a conventional ECN-based, sender-driven AIMD congestion control with receiver-driven fair-share rate hints. Windows are kept per path as well as per connection, so a congestion mark trims the path that saw it and steers the next packets toward paths that are clear. In every acknowledgment, the receiver returns the share of its inbound bandwidth it has allocated to that sender, so senders approach the right speed directly rather than searching for it. Incast resolves in one or two round trips, with better fairness and lower tail latency. MetaRoCE 将传统的基于 ECN 的发送端驱动 AIMD 拥塞控制与接收端驱动的公平份额速率提示相结合。窗口不仅按连接维护，也按路径维护，因此拥塞标记会修剪检测到拥塞的路径，并将后续数据包引导至畅通的路径。在每次确认中，接收端会返回其分配给该发送端的入站带宽份额，因此发送端可以直接达到合适的速率，而无需进行搜索。Incast（多对一通信）问题可以在一到两次往返内解决，具有更好的公平性和更低的尾延迟。

Topology Independence

拓扑无关性

MetaRoCE asks the fabric for two things every switch already has, ECN marking and ECMP. It does not require packet trimming, in-network telemetry, credit-based flow control, or switch-side spraying, and it does not break when a fabric offers them. The same transport runs over fat-tree, multiplane, deep-buffer, and shallow-buffer fabrics, and over vendor clouds whose configuration you don’t control. Nothing proprietary is involved, so the fabric stays free to optimize for cost and cabling. MetaRoCE 对网络架构的要求仅限于每台交换机都已具备的两项功能：ECN 标记和 ECMP。它不需要数据包修剪、带内网络遥测、基于信用的流量控制或交换机端喷洒，并且当网络架构提供这些功能时，它也不会受到影响。相同的传输协议可以在胖树 (fat-tree)、多平面、深缓冲区和浅缓冲区架构上运行，也可以在您无法控制配置的供应商云上运行。它不涉及任何专有技术，因此网络架构可以自由地针对成本和布线进行优化。

Unified Connections at Scale

大规模统一连接

A queue pair (QP) carries both an ordered stream of messages and bandwidth. Traditional RDMA gets more of either by opening more QPs (dozens per node pair), each with a congestion window blind to the rest and its own state on the NIC. MetaRoCE separates the two. A single connection carries many independent ordered streams above, one per communicator or collective, and many paths below, under one congestion controller. The connection state stops growing with the parallelism of the workload. The application layer remains mostly untouched – existing RDMA Verbs APIs and software stacks work without modification. Enhanced features like multiplane support are supported through extension APIs. 队列对 (QP) 同时承载有序的消息流和带宽。传统的 RDMA 通过打开更多的 QP（每节点对数十个）来获取更多带宽或消息流，每个 QP 都有一个对其他 QP 不可见的拥塞窗口，并在网卡上有自己的状态。MetaRoCE 将两者分离开来。单个连接在上方承载许多独立的有序流（每个通信器或集合操作一个），在下方承载许多路径，并由一个拥塞控制器统一管理。连接状态不再随工作负载的并行度而增加。应用层基本保持不变——现有的 RDMA Verbs API 和软件栈无需修改即可工作。多平面支持等增强功能通过扩展 API 提供支持。