本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

Impactful scheduling for GPU clusters

Back to Articles Impactful scheduling for GPU clusters Enterprise Article Published October 9, 2026 Upvote 1 Kyle Wiggers Ai2Comms Follow allenai Building a cluster scheduler to prioritize high-impact research while maintaining full occupancy

在 Ai2 的 AI 基础设施团队，我们负责提供研究所的 GPU 计算能力，专门针对大型分布式训练工作负载。我们将这项任务视为一个由四个相互叠加的指标构成的金字塔。基础是可用性：硬件健康并准备好工作的频率。其上是占用率：分配给特定工作负载的可用时间比例。接下来是影响力：最重要工作负载获得资源分配的频率。金字塔的顶端是利用率：工作负载生命周期内 GPU 容量的使用比例。本文旨在探讨如何提高我们调度决策的影响力。我们最近用一套包含 GPU 时间预算、分层公平份额分配和时间片合约的系统取代了基于优先级的调度程序。结果，我们将关于每个研究项目应获得多少 GPU 时间的争论，从逐案处理的操作任务转变为透明的行政预算流程。

On the AI Infrastructure team at Ai2, we’re responsible for providing the institute’s GPU compute capacity, specifically targeting large, distributed training workloads. We think about this task as a pyramid of four metrics that build on each other. The foundation is availability: how often the hardware is healthy and ready for work. Above this is occupancy: the fraction of available time assigned to a specific workload. Next is impact: how often the most valuable workloads are chosen to receive resources. The capstone of the pyramid is utilization: the fraction of GPU capacity used over the lifetime of a workload. This post is about improving the impact of our scheduling decisions. We recently replaced a priority-based scheduler with a system including GPU time budgets, hierarchical fair-share allocation, and a time-slicing contract. As a result, we shifted the debate about how much GPU time each research project deserves from a case-by-case operational task to a transparent administrative budgeting process.

在 Ai2，我们管理着数千个 NVIDIA H100、B200 和 B300 GPU，它们分布在规模从 88 到 1024 个 GPU 不等的集群中。这些集群专为 AI 模型的大规模分布式训练而构建，服务于约 150 名内部研究人员，他们的工作涵盖了广泛的 AI 领域，包括 LLM 和 VLM 训练的全模型流程、机器人强化学习 (RL) 模拟以及科学智能体用例的后期训练。像许多实验室一样，我们对 GPU 时间的需求远远超过了供应。根据提交的工作负载，在任何时刻，我们对 GPU 的未决请求都比可用数量多出 2-3 倍。可以这样理解：我们集群中每一个可用的 GPU 小时，都有 2-3 个不同的研究工作负载在竞争。

Overcommitting At Ai2, we manage thousands of NVIDIA H100, B200, and B300 GPUs arranged in clusters ranging in size from 88 to 1024 GPUs. These clusters are built for large-scale distributed training of AI models, and they serve a group of about 150 internal researchers whose work covers a diverse set of AI domains, including the full model flow of LLM and VLM training, robotics reinforcement learning (RL) simulation, and post-training for scientific agentic use cases. Like many labs, we have demand for GPU time that far exceeds supply. Based on submitted workloads, at any moment in time we have outstanding requests for 2-3x more GPUs than are available. One way to think about this is that every available GPU hour on our cluster has 2-3 different research workloads competing for it.

从历史上看，我们使用基于优先级的调度程序，并允许工作负载选择不被抢占。每个团队对受保护免于抢占的工作负载所能使用的并发 GPU 数量都有上限。可抢占的工作负载可以在空闲 GPU 上超过该限制。这种策略产生了可预测的病态现象。例如，我们观察到 GPU “蹲位”现象，用户会停放空转的工作负载，以便在需要时连接。这是因为研究人员发现他们无法以足够低的延迟启动调试工作负载来实时解决问题。我们还观察到优先级膨胀，最终 100% 的调度工作负载都使用了“高”优先级。这意味着较低优先级的工作负载完全被剥夺了 GPU 时间。由于抢占是可选的，我们还发现我们的值班工程师大部分工单响应时间都花在协调那些在已知有维护问题的宿主机上运行的不可抢占工作负载的有序关闭上。

Historically, we used a priority-based scheduler, and we allowed workloads to opt out of preemptability. Each team had a limit on concurrent GPUs that could be used by workloads which were protected from preemption. Preemptible workloads could exceed that limit on idle GPUs. This strategy produced predictable pathologies. For example, we observed instances of GPU “squatting” where users would park no-op workloads they could connect to when the need arose. This occurred because researchers found they could not launch debugging workloads with low enough latency to tackle problems in real time. We also observed priority inflation, where eventually 100% of scheduled workloads used HIGH priority. This meant that lower priority levels were starved of GPU time altogether. Since preemptability was optional, we also found that our on-call engineers spent a majority of their ticket response time negotiating the organized shutdown of non-preemptable workloads running on hosts with known maintenance problems.

当这些问题出现时，我们未能及时找出其根本原因。我们最初确保最重要工作获得 GPU 时间的尝试，集中在更严格地控制优先级设置上，并最终通过明确将 GPU 垄断权分配给重要项目来绕过基于优先级的调度程序。虽然起初没有意识到，但我们实际上建立了一个观察“公地悲剧”的完美实验室。个人在竞争稀缺的共享资源，通过寻求个人利益最大化，导致了非最优的全局结果并滥用了底层资源。我们远非第一个观察到这种互动的人。资源分配是一个迷人的研究领域，融合了算法开发、经济学和系统管理。一个核心问题是，用户通常比组织更了解自己工作的价值，但他们可能有动机隐藏这种价值，或者即使在这样做会损害整体性能时也紧抓资源不放。

Tragedy of the commons When these problems emerged, we were slow to identify their root causes. Our initial attempts to ensure the most important work received GPU time were focused on tighter control of how priorities were set, and, ultimately, working around the priority-based scheduler by explicitly assigning GPU monopolies to important projects. While we didn’t recognize it at first, we had built a perfect laboratory for observing the “tragedy of the commons.” Individuals were competing over a scarce, shared resource and, by seeking to maximize individual outcomes, achieving a non-optimal global result and abusing the underlying resource. We were far from the first to observe this kind of interaction. Resource allocation is a fascinating research domain that mixes algorithm development, economics, and system management. A central problem is that users often know the value of their own jobs better than the organization does, but they may have incentives to hide that value or hold on to resources even when doing so hurts total performance.

例如，Ghodsi 等人在 2011 年介绍“主导资源公平性”的论文中讲述了一个轶事：一家搜索公司只有在用户能保证高利用率的情况下，才为工作提供专用机器。他们很快发现，“用户会在代码中加入无限循环，以人为地提高利用率水平。”硬件在变，但使资源分配变得复杂的根本问题依然存在。

For example, in their 2011 paper introducing Dominant Resource Fairness, Ghodsi et al. recount an anecdote in which a search company provided dedicated machines to jobs only if their users could guarantee high utilization. They soon discovered “users would sprinkle their code with infinite loops to artificially inflate utilization levels.” The hardware changes, but the fundamental problems that make resource allocation complex persist.

解决公地悲剧的经典方案是将共享资源私有化——所有者有动力将其财产的价值最大化。当我们为团队分配 GPU 垄断权时，我们已经在做这种方案的一个版本，但它太粗糙了。由于研究的季节性，这导致 GPU 处于闲置状态。团队在不同时间准备运行实验和训练，因此分配垄断权会确保在某些时候没有工作准备执行，而另一个团队却在等待容量。我们是在手动解决背包问题，试图将动态变化的研究需求塞进静态的时间表中。我们想要所有权激励，但也想保持 GPU 的满负荷运行。我们决定迭代所有权模型。我们没有向团队发放 GPU，而是选择分配一部分 GPU 时间。预测未来的需求需要知道新颖科学实验的结果，因此无法精确预测。然而，研究工作的优先级是一个战略问题，可以更容易地提前讨论和决定。我们没有试图解决调度难题，而是让领导层像投资者一样思考。在工作负载存在之前，根据他们对潜在影响力的判断，决定如何用 GPU 时间资助每一项研究工作。调度程序随后可以使用这些信息。

Budgets not schedules The classic solution to a tragedy of the commons is to privatize the shared resource—owners are incentivized to maximize the value of their property. When we assigned teams monopolies over sets of GPUs, we were already doing a version of this, but it was too coarse. It caused GPUs to sit idle due to the seasonality of research. Teams are ready to run experiments and training at different times, so assigning a monopoly would ensure that there would be times when no jobs were ready to execute, with another team left waiting for capacity. We were manually solving a knapsack problem, trying to fit dynamically changing research needs into a static schedule. We wanted the ownership incentive, but we also wanted to maintain full occupancy of the GPUs. We decided to iterate on the ownership model. Instead of issuing teams GPUs, we chose to allocate a portion of GPU time. Predicting demand into the future would require knowing the result of novel science experiments, so it cannot be forecast with precision. Priority across research efforts, however, is a question of strategy, and it can be more easily debated and decided in advance. Instead of trying to solve the scheduling puzzle, we enabled leadership to think like investors. Before the workloads exist, decide how to fund each research effort with GPU time based on their judgment of its likely impact. The scheduler could then use that info