Same Cluster, 33 Points More Utilization: What Changed Was the Order
Same Cluster, 33 Points More Utilization: What Changed Was the Order
同一集群，利用率提升 33 个百分点：改变的只是调度顺序
The previous post argued that utilization, not intelligence, is where the next real constraint in enterprise AI is forming, and it closed by noting that no playbook has emerged yet for what a mature GPU Management practice looks like. This is ours. 上一篇文章指出，企业级 AI 下一个真正的瓶颈在于利用率而非智能，并总结道，目前尚未出现一套成熟的 GPU 管理实践指南。这就是我们的方案。
We built a constraint-aware GPU allocator and benchmarked it against a FIFO scheduler across seven benchmark scenarios. On identical hardware, running identical workloads, GPU utilization rose by as much as 33 percentage points, and priority-weighted output rose in every one of them, by as much as 105%. Nothing about the hardware changed. What changed was the order in which allocation decisions get made. 我们构建了一个具备约束感知能力的 GPU 分配器，并在七个基准测试场景中将其与先进先出（FIFO）调度器进行了对比。在相同的硬件和工作负载下，GPU 利用率最高提升了 33 个百分点，且所有场景下的优先级加权产出均有所增长，最高增幅达 105%。硬件没有任何改变，改变的只是做出分配决策的顺序。
One note on measurement before the numbers start. Every gain below is expressed as improvement over the FIFO result on the same scenario. Utilization is reported in percentage points; value is reported as a percentage increase in priority-weighted output. 在展示数据之前，先说明一下衡量标准。下文中的每一项增益均表示相对于同一场景下 FIFO 结果的提升。利用率以百分点为单位；价值以优先级加权产出的百分比增长来衡量。
The decision, stated precisely “Keep the GPUs busy” is not a decision a system can execute. The decision is narrower and much harder: which GPU runs which job, in which timestep, at what priority. Formally it is one binary choice per combination of GPU, job and timestep, and the output is a grid — every GPU, across the whole scheduling horizon, with a job name in each cell or nothing at all. “让 GPU 保持忙碌”这一决策，系统本身无法直接执行。真正的决策更为具体且困难：哪个 GPU 在哪个时间步长运行哪个任务，以及优先级如何。从形式上讲，这是针对 GPU、任务和时间步长组合的二元选择，输出结果是一个网格——在整个调度周期内，每个 GPU 的每个单元格中要么填入任务名称，要么为空。
Four workload types compete for that grid: training, real-time inference, batch inference, and quantization. They split into two allocation shapes, and the split is where the difficulty lives. Training, batch inference and quantization are batch-like: once started, each needs a contiguous block of GPUs held without interruption until the job finishes. Real-time inference is the opposite: elastic, driven by a demand curve that changes every timestep, growing and shrinking as traffic does. Two incompatible shapes competing for the same hardware in the same timestep is the core problem. A second heterogeneity sits inside a single type: for the same base model, training jobs range from a few hours to several days, and from one GPU to dozens. 四种工作负载类型竞争该网格资源：训练、实时推理、批量推理和量化。它们分为两种分配形态，而这种划分正是难点所在。训练、批量推理和量化属于批处理类：一旦启动，每个任务都需要一块连续的 GPU 资源，且在任务完成前不能中断。实时推理则相反：它是弹性的，由随时间步长变化的需求曲线驱动，随流量波动而伸缩。两种不兼容的形态在同一时间步长竞争同一硬件，这是核心问题。第二种异构性存在于单一类型内部：对于同一个基础模型，训练任务的时长从几小时到几天不等，所需的 GPU 数量也从一个到几十个不等。
What FIFO costs under contention
FIFO 在资源争用下的代价
The comparison point throughout is a FIFO-based scheduler: real-time inference served from a fixed reservation, and every other job placed in arrival order, without regard for priority. Under the right conditions, that is a reasonable policy. When the cluster has slack, allocation order costs nothing in utilization, everything fits regardless of sequence, so FIFO and anything more sophisticated fill the same fraction of the pool. 全程的对比基准是基于 FIFO 的调度器：实时推理通过固定预留资源提供服务，其他所有任务按到达顺序排列，不考虑优先级。在特定条件下，这是一种合理的策略。当集群资源充足时，分配顺序对利用率没有影响，无论顺序如何都能容纳所有任务，因此 FIFO 与更复杂的调度器在资源池填充率上表现一致。
Contention is where that ordering cost stops being invisible and starts costing capacity too. It then becomes expensive in two separate ways, and they are worth taking one at a time. 当出现资源争用时，这种排序带来的代价就不再隐形，并开始损耗容量。这会从两个方面带来高昂成本，我们逐一分析。
The reservation. Real-time inference cannot wait for capacity; the GPUs have to be there the moment traffic needs them. A scheduler that places jobs in arrival order has no mechanism for releasing GPUs during a trough and reclaiming them before the next peak, so the only way to guarantee availability is to take each real-time application’s maximum demand for the day and reserve that many GPUs for the whole day. The cost lands in every hour that is not the peak. An application needing six GPUs at midday and two at 4am holds all six for twenty-four hours, and the four idle GPUs are unavailable to any batch job for the entire day. They are not being used, and they are not free either. It is why the baseline sits near half the cluster in the two scenarios where reservation dominates: 51.6% in the mixed control and 53.6% in the training-heavy case. Roughly half a pool, with much of the idle half reserved rather than free. This cost is paid whether the cluster is contended or not — contention only makes it visible. 预留机制。实时推理无法等待资源；流量到来时 GPU 必须立即可用。按到达顺序分配任务的调度器没有在低谷期释放 GPU 并在高峰前回收的机制，因此保证可用性的唯一方法是按实时应用全天最高需求预留 GPU。这种成本体现在非高峰期的每一个小时。一个中午需要 6 个 GPU、凌晨 4 点只需 2 个的应用会占用全部 6 个 GPU 长达 24 小时，而那 4 个空闲的 GPU 全天都无法被任何批处理任务使用。它们既没被使用，也不处于空闲状态。这就是为什么在预留占主导的两个场景中，基准利用率徘徊在集群容量的一半左右：混合控制场景为 51.6%，训练密集型场景为 53.6%。大约一半的资源池中，大部分空闲部分是被预留而非真正空闲。无论集群是否处于争用状态，这种成本都在支付——争用只是让它变得显而易见。
The ordering. Under real contention, which jobs fit at all depends on the order you place them, not just on how much capacity exists. Order is not a tiebreaker applied after the capacity question is settled. Order is a capacity decision. FIFO places each job as it arrives, without weighing what that job is worth and without checking what else still has to fit inside the horizon, so high-priority work waits behind whatever asked first and capacity gets committed in placements that later jobs cannot use. 排序机制。在真正的资源争用下，哪些任务能被执行取决于你安排它们的顺序，而不仅仅取决于有多少容量。顺序不是在解决容量问题后才使用的平局决胜机制，顺序本身就是一种容量决策。FIFO 按任务到达顺序进行分配，不权衡任务价值，也不检查调度周期内还有哪些任务需要容纳，因此高优先级任务必须排在先到达的任务之后，导致容量被占用在后续任务无法利用的位置上。
The two compound. The block held for the day’s maximum real-time demand is off the table for every batch job in the queue, in every hour, and whatever remains is handed out in the order the requests happened to arrive. It is the GPU equivalent of an airline assigning aircraft to whichever charter called first, then finding nothing left to fly the route that actually pays. And GPUs reserved all day for a peak lasting a couple of hours are the grounded aircraft from the previous piece in the most literal sense: on standby, earning nothing, unavailable to anyone else. 两者叠加。为全天实时需求峰值预留的资源块在每一小时内都对队列中的所有批处理任务不可用，而剩余资源则按请求到达顺序分配。这相当于航空公司将飞机分配给第一个打来电话的包机客户，结果发现没有飞机去飞真正赚钱的航线。而那些为几小时高峰期预留了一整天的 GPU，从字面上看就是上一篇文章中提到的“停飞飞机”：处于待命状态，不产生任何收益，且对其他人不可用。
Across five benchmark scenarios built for genuine contention, the allocator improved both axes at once. Utilization moved from a 52–85% band to a 72–88% band. Priority-weighted value rose between 24.6% and 105.1%, averaging 52%. Every scenario, both metrics, no tradeoff to explain away. The strongest single case was a training-heavy workload on 8 GPUs: utilization went from 53.6% to 87.0%, and value more than doubled, up 105%. Thirty-three points of a fixed, already-depreciating asset, recovered by reclaiming reserved standby capacity and placing the rest in priority order. (This figure reflect a single baseline ordering.) The allocator removes both behaviors. Real-time demand is treated as a curve rather than a ceiling, allocated 在五个针对真实争用构建的基准测试场景中，该分配器同时提升了两个维度。利用率从 52–85% 的区间提升至 72–88%。优先级加权产出增长了 24.6% 到 105.1%，平均增幅为 52%。每个场景、两项指标均有提升，无需任何权衡解释。最显著的案例是 8 个 GPU 上的训练密集型工作负载：利用率从 53.6% 提升至 87.0%，产出翻了一倍多，增长了 105%。通过回收预留的待命容量并按优先级顺序安排其余任务，我们找回了 33 个百分点的固定资产利用率，而这些资产原本正在不断贬值。（此数据反映了单一基准排序的结果。）该分配器消除了上述两种行为。它将实时需求视为一条曲线而非上限进行分配。