From User Sequences to Scaling Laws: A Multi-Stage Architecture for Meta’s Ads Ranking
From User Sequences to Scaling Laws: A Multi-Stage Architecture for Meta’s Ads Ranking
从用户序列到缩放定律：Meta 广告排序的多阶段架构
By Steven De Gryze, Parshva Doshi, Sean O’Byrne, Arnold Overwijk, Dinesh Ramasamy, Lee Xiong 作者：Steven De Gryze, Parshva Doshi, Sean O’Byrne, Arnold Overwijk, Dinesh Ramasamy, Lee Xiong
Every day, Meta’s recommendation platforms handle billions of user interactions, generating rich temporal signals that capture individual preferences and intent across products, ads, and content. Meta 的推荐平台每天处理数十亿次用户交互，生成丰富的时序信号，捕捉用户在产品、广告和内容方面的个人偏好与意图。
In our 2024 post on sequence learning for ads recommendations, we showed how modeling the order and timing of user actions (rather than relying on static, manually engineered sparse features) produces richer, sequence-aware representations of user interests and ad preferences. 在我们 2024 年关于广告推荐序列学习的文章中，我们展示了如何通过建模用户行为的顺序和时间（而不是依赖静态、人工设计的稀疏特征）来生成更丰富、具备序列感知能力的用户兴趣和广告偏好表示。
This post goes a step further, introducing two architectural breakthroughs that let us scale sequence learning advancements from foundational innovations into a production platform with predictable, LLM-style scaling laws: (1) a multi-stage sequence model that decouples heavy offline user modeling from lightweight online ranking tasks and (2) a learning technique based on dense tokenization and target-aware attention that efficiently learns feature interactions directly from data. 本文更进一步，介绍了两项架构突破，使我们能够将序列学习的进展从基础创新扩展到具有可预测、类 LLM 缩放定律的生产平台：（1）一种将繁重的离线用户建模与轻量级在线排序任务解耦的多阶段序列模型；（2）一种基于密集标记化（dense tokenization）和目标感知注意力（target-aware attention）的学习技术，能够直接从数据中高效学习特征交互。
Together with our broader model innovations, these advancements have contributed to a cumulative lift of 6% in conversions on Instagram, 3% in conversions on Facebook and 3.5% in ad clicks on Facebook. 结合我们更广泛的模型创新，这些进展促使 Instagram 的转化率累计提升了 6%，Facebook 的转化率提升了 3%，Facebook 的广告点击率提升了 3.5%。
This unified platform for sequence modeling is a core component of Meta’s Generative Ads Recommendation Model (GEM), helps to harness the comprehensive user behavioral understanding of this learning paradigm to maximize the benefit to advertisers. 这一统一的序列建模平台是 Meta 生成式广告推荐模型（GEM）的核心组件，有助于利用该学习范式对用户行为的全面理解，从而最大化广告主的收益。
The Historical Challenges of Sequence Modeling
序列建模的历史挑战
Ads recommendation systems must retrieve and rank thousands of ads within milliseconds, processing millions of candidates per second. To manage this scale, some approaches to sequence models rely on hybrid model configurations where a specific model processes user event sequences and another model handles sparse feature interactions. 广告推荐系统必须在毫秒级时间内检索并排序数千条广告，每秒处理数百万个候选对象。为了应对这种规模，一些序列模型方法依赖混合模型配置，即由一个特定模型处理用户事件序列，另一个模型处理稀疏特征交互。
While effective at meeting production demands, this hybrid approach has potential tradeoffs: Lossy knowledge transfer between components, Continued reliance on manual feature engineering, Scaling ceilings from interference between ranking and sequence model components. 虽然这种混合方法在满足生产需求方面行之有效，但存在潜在的权衡：组件间有损的知识传递、对人工特征工程的持续依赖，以及排序与序列模型组件之间干扰导致的扩展瓶颈。
Scaling both temporal sequence lengths and the transformer models that process them can turn the tradeoffs of the hybrid approach into a bottleneck, limiting the ability to improve the ads experience of users and the performance of advertisers’ campaigns. 同时扩展时间序列长度和处理它们的 Transformer 模型，可能会使混合方法的权衡变成瓶颈，从而限制了改善用户广告体验和提升广告主投放效果的能力。
We’ve made two fundamental architectural breakthroughs in sequence learning that resolve the core tension between model complexity and serving efficiency: (1) a multi-stage sequence model that decouples offline user modeling from online ranking and (2) a dense tokenization with target-aware attention learning paradigm. 我们在序列学习方面取得了两个根本性的架构突破，解决了模型复杂性与服务效率之间的核心矛盾：（1）将离线用户建模与在线排序解耦的多阶段序列模型；（2）带有目标感知注意力的密集标记化学习范式。
Together, they provide a flexible production strategy that helps generalize sequence learning models and establish an LLM-style scaling law that predictably balances model performance with compute. 它们共同提供了一种灵活的生产策略，有助于推广序列学习模型，并建立一种可预测地平衡模型性能与计算资源的类 LLM 缩放定律。
Introducing the Multi-Stage Sequence Model
引入多阶段序列模型
To address scaling efficiency, a multi-stage model has been developed that enables scaling of a transformer-based sequence model in a compute efficient manner. Separating the sequence model into two complementary stages (upstream/offline user modeling and downstream/online ranking), enables model capacity to scale so that performance keeps improving without proportional increases in serving resources. 为了解决扩展效率问题，我们开发了一种多阶段模型，能够以计算高效的方式扩展基于 Transformer 的序列模型。将序列模型分为两个互补阶段（上游/离线用户建模和下游/在线排序），使得模型容量能够扩展，从而在不按比例增加服务资源的情况下持续提升性能。
In Figure 1, the left panel shows the offline user model. It processes long user histories asynchronously and produces cached embeddings that capture deep behavioral patterns. The right panel shows the online ranking model that combines these cached representations with real time ad candidate signals to produce the final ranking. The arrow between the two stages carries the user feature embeddings from offline → online ranking models. 在图 1 中，左侧面板显示了离线用户模型。它异步处理长用户历史记录，并生成捕捉深层行为模式的缓存嵌入。右侧面板显示了在线排序模型，它将这些缓存的表示与实时广告候选信号相结合，产生最终排序。两个阶段之间的箭头表示用户特征嵌入从离线模型流向在线排序模型。
(Figure 1: An overview of the multi-stage model.) （图 1：多阶段模型概览。）
Two Key Stages of the Model
模型的两个关键阶段
First Stage: Offline User Model 第一阶段：离线用户模型
User-side features are processed asynchronously using deep transformer upstream models. These models scale to several transformer layers with sequence lengths in the thousands and generate embeddings that are precomputed and cached at the user level. The upstream model strictly separates user features from ad and context features to ensure user embeddings remain independent of any particular ad candidate. 用户侧特征使用深度 Transformer 上游模型进行异步处理。这些模型可扩展至多个 Transformer 层，序列长度达数千，并生成在用户层面预计算和缓存的嵌入。上游模型严格将用户特征与广告及上下文特征分离，以确保用户嵌入独立于任何特定的广告候选对象。
Second Stage: Online Ranking Model 第二阶段：在线排序模型
The offline user model representations are complemented with online ranking models that use fresh user signals and ad candidate information for real time ranking. This stage is optimized for speed, meeting strict latency budgets while leveraging the deep representations computed offline. 离线用户模型的表示由在线排序模型补充，后者使用最新的用户信号和广告候选信息进行实时排序。该阶段针对速度进行了优化，在满足严格延迟预算的同时，利用离线计算出的深层表示。
Separating the sequence modeling system into two distinct, yet complementary, stages enables an increase in model complexity along a scaling curve for the Offline User Model without causing a spike in spike in serving costs for the Online Ranking Models. 将序列建模系统分为两个独特但互补的阶段，使得离线用户模型能够沿着缩放曲线增加模型复杂性，而不会导致在线排序模型的服务成本激增。
Sequence Model Architecture Innovations
序列模型架构创新
Dense Tokenization 密集标记化
This tokenization approach integrates sparse features with sequential behavioral data into a single dense vocabulary, enabling attention mechanisms to discover interactions independently. Unlike traditional recommendation systems, which rely on manually engineered representations to capture sparse cross-feature interactions, this approach lets the model learn those interactions directly from the data. 这种标记化方法将稀疏特征与序列行为数据整合到一个单一的密集词汇表中，使注意力机制能够独立发现交互。与依赖人工工程表示来捕捉稀疏交叉特征交互的传统推荐系统不同，这种方法让模型直接从数据中学习这些交互。
Target-Aware Multi-Head Attention 目标感知多头注意力
Tokenized sparse features and ad candidate information are fused with user behavior sequences, then processed by a memory-efficient form of multi-head attention that lets each layer weigh a user’s past behaviors against the specific ad being scored. 标记化后的稀疏特征和广告候选信息与用户行为序列融合，然后由一种内存高效的多头注意力形式进行处理，使每一层都能根据正在评分的具体广告来权衡用户的过去行为。
Stacking multiple aligned attention blocks with stable attention distributions allows each layer to capture higher-order interactions between the target ad and the user’s historical behavior, progressively distilling long sequences into compact representations. 堆叠多个具有稳定注意力分布的对齐注意力块，使每一层都能捕捉目标广告与用户历史行为之间的高阶交互，逐步将长序列提炼为紧凑的表示。
A Predictable Scaling Curve
可预测的缩放曲线
LLM-Style Scaling Law 类 LLM 缩放定律
When running on real-world ads traffic… 当在真实的广告流量上运行时……