GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model

GEM 训练：Meta 如何将其广告基础大模型的训练效率提升一倍

By Darren Liu, Huayu Li, Raghav Boinepalli, Yuzhen Huang, Jackie (Jiaqi) Xu, Richard Qiu, Chunzhi Yang, Rich Zhu, Dev (Devashish) Shankar, Huaqing Xiong, Lei Tian, Ruilin Chen, Xiaoyi (Leo) Liu, Yasmine Badr, Wentao Duan

Meta’s Generative Ads Recommendation Model (GEM), the foundation model behind ads recommendations across Instagram and Facebook, now trains at LLM scale on several thousand of the latest-generation GPUs. This post goes into the details on how we achieved: doubling end-to-end (E2E) training efficiency to 20–25% Model FLOPs Utilization (MFU) while scaling training FLOPs 4x in 12 months, by co-designing kernels, precision, parallelism, networking, and memory together.

Meta 的生成式广告推荐模型 (GEM) 是 Instagram 和 Facebook 广告推荐背后的基础模型，目前已在数千个最新一代 GPU 上以大模型 (LLM) 规模进行训练。本文详细介绍了我们如何通过协同设计内核、精度、并行性、网络和内存，在 12 个月内将训练 FLOPs 扩展 4 倍的同时，将端到端 (E2E) 训练效率提升至 20–25% 的模型浮点运算利用率 (MFU)。

Training GEM presents unique engineering challenges at the intersection of recommendation systems and LLMs as the model combines a hybrid architecture plus recommendations-domain data properties that are unlike typical LLM workloads. AI infrastructure optimized for LLM training (kernels, parallelism, low precision recipes etc.) does not directly transfer, requiring significant innovation and hardware/software co-design to reach LLM-scale training for recommendation models efficiently.

训练 GEM 在推荐系统与大模型的交叉领域提出了独特的工程挑战，因为该模型结合了混合架构以及与典型大模型工作负载不同的推荐领域数据特性。针对大模型训练优化的 AI 基础设施（内核、并行性、低精度方案等）无法直接迁移，需要重大的创新和软硬件协同设计，才能高效实现推荐模型的大规模训练。

We tackled these challenges through complementary compute efficiency and scaling efficiency innovations: Compute efficiency: Achieved through a customized recommendation kernel library — Jagged Flash Attention (JFA), Generalized Dot-Product Attention (GDPA), BlockAttention, etc. — and mixed ultra-low precision training (including MXFP8 attention and MLP) optimized for recommendation workloads, purpose-built to exploit latest generation GPU’s architecture. Scaling efficiency: Topology-aware 5D parallelism with Streaming Multiprocessor (SM)-free collectives — 2D FSDP + Expert Parallelism for dense parameters, combined with Fully Sharded 2D Model Parallelism for sparse parameters — co-designed with Meta’s multi-tiered network hierarchy to reduce communication overhead.

我们通过计算效率和扩展效率方面的互补创新解决了这些挑战： 计算效率：通过定制的推荐内核库（如 Jagged Flash Attention (JFA)、广义点积注意力 (GDPA)、BlockAttention 等）以及针对推荐工作负载优化的混合超低精度训练（包括 MXFP8 注意力和 MLP）实现，这些技术专为利用最新一代 GPU 架构而构建。 扩展效率：采用具有流式多处理器 (SM) 无关集合通信的拓扑感知 5D 并行化——针对稠密参数的 2D FSDP + 专家并行，结合针对稀疏参数的完全分片 2D 模型并行——并与 Meta 的多层网络架构协同设计，以减少通信开销。

The results: we doubled GEM’s E2E training efficiency to 20-25% MFU while scaling total training FLOPs 4x over the past 12 months.

结果：在过去 12 个月中，我们将 GEM 的端到端训练效率提高了一倍，达到 20-25% 的 MFU，同时将总训练 FLOPs 扩展了 4 倍。

GEM’s Architecture And Its Unique Training Challenges

GEM 的架构及其独特的训练挑战

GEM is the central recommendations foundation model behind Meta’s ads system. It has a hybrid architecture with trillions of sparse embedding parameters and billions of dense parameters. GEM is trained on ad content and user engagement data with two categories of features: sequence features (e.g., user activity history) and non-sequence features (e.g., user location, ad creative representation). Customized attention mechanisms are applied to each group independently, while also enabling cross-feature learning. The interplay between this hybrid architecture and rec-domain data properties is what makes GEM’s training uniquely challenging.

GEM 是 Meta 广告系统背后的核心推荐基础模型。它采用混合架构，拥有数万亿个稀疏嵌入参数和数十亿个稠密参数。GEM 基于广告内容和用户参与数据进行训练，这些数据包含两类特征：序列特征（如用户活动历史）和非序列特征（如用户位置、广告创意表示）。定制的注意力机制被独立应用于每一组特征，同时支持跨特征学习。这种混合架构与推荐领域数据特性之间的相互作用，使得 GEM 的训练极具挑战性。

Challenge 1: Achieving High Per-GPU Utilization

挑战 1：实现高单 GPU 利用率

Today’s data center GPUs and their software stacks are mostly optimized for LLM workloads, whereas recommendation workloads have a fundamentally different profile due to unique data characteristics and rich user & ads signal interaction patterns that make it extremely difficult to achieve high GPU compute utilization for training a foundational recommendation model of GEM’s size.

当今的数据中心 GPU 及其软件栈大多针对大模型工作负载进行了优化，而推荐工作负载由于其独特的数据特性以及丰富的用户与广告信号交互模式，具有根本不同的特征，这使得在训练 GEM 这种规模的基础推荐模型时，极难实现高 GPU 计算利用率。

• Jagged Inputs: Training samples have highly variable sequence length as user activity history can vary wildly. Padding to max length would waste up to 50% compute.

• Diverse interaction patterns and asymmetric sequences: Self-attention operates on extremely long sequences (activity history) but short attention window; cross-attention learns user x ads interaction with long queries but short key/value; pooled multi-head attention (PMA) compress user activity history, resulting in short queries but long key/value. These asymmetric shapes make intra kernel pipelining less effective to saturate compute units.

• Memory-bound operations: e.g., small embedding dimension for MLP and various normalizations for model quality and training stability leave compute units underutilized.

• Numerical sensitivity: Ads optimization tasks (CTR/CVR prediction) are highly sensitive to numerical change (e.g., precision), making naïve low-precision training prone to quality regression.

• 锯齿状输入 (Jagged Inputs)： 训练样本的序列长度差异巨大，因为用户活动历史变化剧烈。填充至最大长度会浪费高达 50% 的计算资源。

• 多样的交互模式和非对称序列： 自注意力机制在极长的序列（活动历史）上运行，但注意力窗口较短；交叉注意力机制学习用户与广告的交互，查询 (Query) 较长但键/值 (Key/Value) 较短；池化多头注意力 (PMA) 压缩用户活动历史，导致查询较短但键/值较长。这些非对称形状使得内核内流水线难以充分利用计算单元。

• 内存受限操作： 例如，MLP 的小嵌入维度以及为了模型质量和训练稳定性而进行的各种归一化操作，导致计算单元利用率不足。

• 数值敏感性： 广告优化任务（CTR/CVR 预测）对数值变化（如精度）高度敏感，使得简单的低精度训练容易导致质量下降。

Challenge 2: Scaling Efficiently Across Thousands of GPUs

挑战 2：在数千个 GPU 上高效扩展

Training GEM across thousands of GPUs with trillions of sparse embedding parameters and billions of dense parameters requires scaling efficiently, not just scaling up. Simply adding more GPUs does not translate to proportional speedup. In distributed training, E2E latency per training step is determined by: E2E Latency = Max across GPU Rank (Max(Local Compute Time, Communication Time)).

在拥有数万亿稀疏嵌入参数和数十亿稠密参数的数千个 GPU 上训练 GEM，需要的是高效扩展，而不仅仅是规模扩大。简单地增加 GPU 数量并不能带来成比例的加速。在分布式训练中，每个训练步骤的端到端延迟由以下公式决定：端到端延迟 = GPU 秩中的最大值 (最大(本地计算时间, 通信时间))。

Near-linear scaling requires four conditions: Total compute time >> total communication time; Communication hidden behind compute without contention; Minimal recomputation from memory pressure; Good load balancing across ranks. GEM’s workload threatens every one of these.

近线性扩展需要满足四个条件：总计算时间 >> 总通信时间；通信隐藏在计算之后且无竞争；内存压力导致的重计算最小化；各秩之间的负载均衡良好。GEM 的工作负载对上述每一项都构成了挑战。

Our Approach and Efficiency Framework

我们的方法与效率框架

Given the challenges outlined above, we needed a framework that turned a sprawling co-design effort into a small number of technical levers. We measure training efficiency through E2E MFU, which decomposes into two factors: E2E MFU = Local MFU (compute efficiency) × Scaling Ratio (scaling efficiency). These factors describe two related but distinct optimization problems. Local MFU (compute efficiency) measures how well a single GPU’s compute units are utilized — how close the workload runs to the hardware roofline. It is determined by kernel design.

鉴于上述挑战，我们需要一个框架，将庞大的协同设计工作转化为少数几个技术杠杆。我们通过端到端 MFU 来衡量训练效率，它分解为两个因素：端到端 MFU = 本地 MFU（计算效率）× 扩展比率（扩展效率）。这些因素描述了两个相关但不同的优化问题。本地 MFU（计算效率）衡量单个 GPU 计算单元的利用程度——即工作负载运行接近硬件性能上限的程度。它由内核设计决定。