Modernizing the Meta Ads Service With an Open-Source Kernel Scheduler
Modernizing the Meta Ads Service With an Open-Source Kernel Scheduler
利用开源内核调度器实现 Meta 广告服务的现代化
By Blaise Sanouillet, Sunyi Shao, Jeff Song, Gosh Arzumanyan, Shawn Wu, Tejun Heo, Praveen Sampath, GP Musumeci 作者：Blaise Sanouillet, Sunyi Shao, Jeff Song, Gosh Arzumanyan, Shawn Wu, Tejun Heo, Praveen Sampath, GP Musumeci
TL; DR At Meta’s scale, a few milliseconds of latency degradation can have a significant negative impact on ads performance. When a Linux kernel upgrade risked regressing latency across Meta’s ad serving fleet, we turned to sched_ext — the upstream, BPF-based extensible scheduling framework — to build a scheduling policy customized to the Ads delivery workload. The result: a 28% reduction in ads retrieval stage tail(99th percentile) latency, 3.28 megawatts(MW) power saving, and a 1.1% increase in the number of ads ranked, proving that workload-specific scheduling optimization can directly drive business value. 简而言之：在 Meta 的规模下，几毫秒的延迟退化都可能对广告性能产生重大负面影响。当一次 Linux 内核升级可能导致 Meta 广告服务集群的延迟回退时，我们转向了 sched_ext（一种基于 BPF 的上游可扩展调度框架），旨在构建一套针对广告投放工作负载定制的调度策略。结果显示：广告检索阶段的尾部延迟（99 分位）降低了 28%，节省了 3.28 兆瓦（MW）的电力，广告排名数量增加了 1.1%。这证明了针对特定工作负载的调度优化可以直接驱动业务价值。
Why Ads Latency Matters 为什么广告延迟至关重要
Meta’s ads serving fleet handles more than 5 million requests per second on average at the serving platform entry point, which is over 400 billion per day across all monetized surfaces. Every millisecond shaved off the p99 latency makes the ads more relevant for people on our platforms, and better matches mean stronger ROI for advertisers. This provides a real opportunity to reduce latency through workload-specific scheduling. That is why our Ads and Linux Kernel teams have been working together to build a scheduling policy customized to the ads delivery workload using sched_ext, the upstream, BPF-based extensible scheduling framework. Meta 的广告服务集群在服务平台入口处平均每秒处理超过 500 万次请求，在所有商业化界面上每天处理超过 4000 亿次请求。p99 延迟每减少一毫秒，都能让广告对我们平台上的用户更具相关性，而更好的匹配意味着广告商能获得更强的投资回报率（ROI）。这为通过特定工作负载调度来降低延迟提供了真正的机会。因此，我们的广告团队和 Linux 内核团队一直在合作，利用 sched_ext（一种基于 BPF 的上游可扩展调度框架）构建一套针对广告投放工作负载定制的调度策略。
Until now, we have been using the general-purpose schedulers typically integrated in the Linux kernel (CFS and EEVDF) that balance threads across CPUs with no understanding of the workload. However, here we know the purpose and importance of each thread. With sched_ext, we can encode this knowledge directly into the scheduler. Work that improves the p99 request latency is scheduled first, and everything else takes a back seat. 在此之前，我们一直使用 Linux 内核中常见的通用调度器（CFS 和 EEVDF），它们在 CPU 之间平衡线程时并不了解工作负载的特性。然而，在这里我们清楚每个线程的目的和重要性。通过 sched_ext，我们可以将这些知识直接编码到调度器中。能够改善 p99 请求延迟的工作会被优先调度，而其他任务则退居次要位置。
sched_ext at Meta Meta 的 sched_ext
sched_ext is an open-source, BPF-based scheduler framework that officially entered kernel v6.12. We developed it by partnering with the authors of Google’s ghOSt to design a scheduler suitable for upstream Linux integration. It has already been deployed in several services at Meta, delivering meaningful reductions in scheduling latency. While upgrading our fleet to the latest stable version of Linux (kernel v6.9) we observed that the new Earliest Eligible Virtual Deadline First (EEVDF) scheduler introduced in Linux kernel v6.6 was causing a latency regression which reduced the number of ads ranked in response. As a result, a subset of ads hosts were forced to remain on the older v6.4 kernel, creating technical debt and operational fragmentation. Given its already strong performance, sched_ext was a great candidate to address these scheduling regressions. sched_ext 是一个基于 BPF 的开源调度器框架，已正式进入 Linux 内核 v6.12 版本。我们通过与 Google ghOSt 的作者合作开发了它，旨在设计一种适合集成到 Linux 上游的调度器。它已在 Meta 的多个服务中部署，显著降低了调度延迟。在将集群升级到最新的 Linux 稳定版本（内核 v6.9）时，我们观察到 Linux 内核 v6.6 中引入的全新“最早合格虚拟截止时间优先”（EEVDF）调度器导致了延迟回退，从而减少了响应中广告的排名数量。因此，部分广告主机被迫保留在较旧的 v6.4 内核上，这造成了技术债务和运维碎片化。鉴于其强大的性能，sched_ext 成为了解决这些调度回退问题的绝佳方案。
Custom Scheduling with sched_ext 使用 sched_ext 进行自定义调度
Sched_ext lets scheduler developers implement their preferred scheduling policy as a BPF program. When a host starts running the ads workload, an ads-optimized policy is applied. From that point on, the kernel calls into the BPF scheduler through a set of event-driven callbacks to handle common scheduling events, such as: Thread wake-up: choose a CPU when a thread becomes runnable. Enqueue: place a thread in a run queue. Dispatch: select the next thread when a CPU becomes idle. Idle transitions: respond to CPUs entering/leaving idle states. Sched_ext 允许调度器开发者将他们偏好的调度策略实现为 BPF 程序。当主机开始运行广告工作负载时，系统会应用针对广告优化的策略。从那时起，内核通过一组事件驱动的回调函数调用 BPF 调度器，以处理常见的调度事件，例如：线程唤醒（当线程变为可运行时选择 CPU）、入队（将线程放入运行队列）、分发（当 CPU 空闲时选择下一个线程）、空闲转换（响应 CPU 进入/离开空闲状态）。
At a high level, the policy soft-partitions CPUs into two pools, one for threads on the latency-critical request path and one for less latency-sensitive work. Which thread goes into which pool is part of the domain-specific knowledge encoded inside the policy. The size of each pool is adjusted dynamically using load-based heuristics. This approach tends to keep related work on the same CPUs over time, improving last-level cache (L3) locality and reducing costly DRAM access. The policy is packaged as a user-space binary that loads the BPF program. That design makes experimentation and performance optimization much faster. To roll out a change, we can simply restart the scheduler process to unload the old policy and load the new one, without rebuilding or reinstalling the kernel. 从宏观上看，该策略将 CPU 软分区为两个池：一个用于延迟敏感的请求路径上的线程，另一个用于对延迟不那么敏感的工作。哪些线程进入哪个池，属于策略中编码的领域特定知识的一部分。每个池的大小通过基于负载的启发式算法动态调整。这种方法倾向于让相关工作长期保持在相同的 CPU 上，从而改善末级缓存（L3）的局部性并减少昂贵的 DRAM 访问。该策略被打包为一个加载 BPF 程序的用户空间二进制文件。这种设计使得实验和性能优化变得更快。要发布更改，我们只需重启调度器进程以卸载旧策略并加载新策略，而无需重新构建或重新安装内核。
Results and Impact 结果与影响
The initial launch took place to switch from kernel 6.4 with the CFS scheduler to kernel 6.9 with sched_ext on the largest ads serving server type. Based on the backtest experiment, the launch delivered: +1.1% on weighted-ads-ranked (metric for number of ads retrieved and ranked). 3.28 megawatts of power savings across the fleet. 28% reduction in service p99 latency on the ads retrieval path. 最初的发布是在最大的广告服务服务器类型上，从使用 CFS 调度器的内核 6.4 切换到使用 sched_ext 的内核 6.9。根据回测实验，此次发布实现了：加权广告排名（衡量检索和排名广告数量的指标）提升 1.1%；整个集群节省 3.28 兆瓦电力；广告检索路径上的服务 p99 延迟降低 28%。
Compounding improvements. Two follow-on scheduler-policy updates, delivered as purely user-space changes, extended the win: Additional 60% reduction in service p99 latency. 18% reduction in timeout errors on the critical path. This is a non-trivial win delivered with no dependency on kernel releases. Each follow-on iteration above shipped in days rather than months because the scheduler policy lives in user space as a BPF program. That cadence is what turned sched_ext from a “kernel upgrade unblocker” to a continuous-optimization platform for ads serving. 持续的改进。两次后续的调度器策略更新（作为纯用户空间更改交付）进一步扩大了成果：服务 p99 延迟额外降低 60%；关键路径上的超时错误减少 18%。这是一个无需依赖内核发布即可实现的重大胜利。上述每次后续迭代都在几天内完成，而不是几个月，因为调度器策略作为 BPF 程序存在于用户空间中。正是这种节奏将 sched_ext 从一个“内核升级的阻碍解除工具”转变为广告服务的持续优化平台。
From Short-Term Fix to Strategic Asset 从短期修复到战略资产
What started out as a targeted response to a very specific operational issue has turned out to be much more strategic, and widely applicable, than we originally anticipated. sched_ext delivers some key benefits to Meta: A parallel and decoupled scheduler optimization path. Upstream Linux scheduling naturally evolves over time, sometimes in larger steps (such as the CFS-to-EEVDF transition), which can be disruptive to downstream consumers. sched_ext gives Meta the flexibility to continuously improve these custom schedulers alongside that evolution. We run and refine our own BPF-based scheduling logic, tailored to the unique demands of our production workloads, so our critical services stay optimized regardless of what happens upstream. Independent deployment and reduced overheads. Scheduler improvements ship as BPF program updates, shipped in days rather than months. 最初只是针对特定运维问题的定向响应，结果却比我们预期的更具战略意义且应用更广泛。sched_ext 为 Meta 带来了几个关键优势：并行且解耦的调度器优化路径。上游 Linux 调度随着时间的推移自然演进，有时步子很大（例如从 CFS 到 EEVDF 的过渡），这可能会对下游用户造成干扰。sched_ext 让 Meta 有灵活性在演进过程中持续改进这些自定义调度器。我们运行并完善自己的基于 BPF 的调度逻辑，针对生产工作负载的独特需求进行定制，因此无论上游发生什么，我们的关键服务都能保持优化。独立部署和降低开销。调度器的改进以 BPF 程序更新的形式发布，在几天内即可交付，而非几个月。