Calibrate, Then Route: A Measured Study of Learned Request Routing for Disaggregated LLM Serving

先校准，后路由：针对解耦大模型服务中学习型请求路由的实证研究

Abstract: Disaggregated LLM serving places compute-heavy prefill and memory-heavy decode on separate GPU pools. Systems such as DistServe, Splitwise, and Mooncake make this separation fast, but routing still determines which instances handle each request. We study a router that estimates the additional completion time on each instance using exact prompt length, predicted output length, post-admission KV cache pressure, and SLO class.

摘要： 解耦大模型（LLM）服务将计算密集型的预填充（prefill）和内存密集型的解码（decode）任务放置在不同的 GPU 池中。DistServe、Splitwise 和 Mooncake 等系统使得这种分离变得高效，但路由策略仍然决定了每个请求由哪个实例处理。我们研究了一种路由算法，该算法通过精确的提示词长度、预测的输出长度、准入后的 KV 缓存压力以及服务等级目标（SLO）类别，来估算每个实例的额外完成时间。

We develop the policy in a discrete event simulator and validate it on eight NVIDIA A40 GPUs, each running a vLLM engine, with NIXL transferring KV caches between pools. All workloads run at measured saturation. Across three mixed, bursty arrival traces, the calibrated router achieves the highest mean goodput at 0.864, compared with 0.835 to 0.847 for round robin, least loaded, and a length heuristic. It also shows the lowest variance across traces.

我们在离散事件模拟器中开发了该策略，并在八块 NVIDIA A40 GPU 上进行了验证，每块 GPU 运行一个 vLLM 引擎，并使用 NIXL 在池之间传输 KV 缓存。所有工作负载均在测定的饱和度下运行。在三个混合且具有突发性的到达轨迹测试中，经过校准的路由器的平均有效吞吐量（goodput）最高，达到 0.864，而轮询（round robin）、最少负载（least loaded）和长度启发式算法的吞吐量在 0.835 到 0.847 之间。它在不同轨迹间的方差也表现最低。

It beats round robin and the length heuristic on all three traces and least loaded on two. On the third, it trails by 0.003, within run to run noise. Hardware calibration matters: simulator-derived constants cost 4.5 goodput points and roughly 40 percent of the tail latency advantage, reducing the scorer to little more than queue counting. Benefits grow with decode pool size and traffic heterogeneity but disappear in pools with three instances, where queue counts are often enough.

它在所有三个轨迹测试中均优于轮询和长度启发式算法，并在其中两个测试中优于最少负载算法。在第三个测试中，它仅落后 0.003，处于运行噪声范围内。硬件校准至关重要：使用模拟器导出的常数会导致有效吞吐量下降 4.5 个点，并损失约 40% 的尾部延迟优势，使评分器退化为仅相当于队列计数。这种优势随着解码池规模和流量异构性的增加而增长，但在仅有三个实例的池中会消失，因为此时简单的队列计数往往就足够了。

Under extreme scarcity, greedy cost minimization concentrates requests on the cheapest scored instance, and blind spreading performs better. With calibrated costs, the learned router matches the goodput of round robin using six GPUs instead of seven.

在资源极度匮乏的情况下，贪婪的成本最小化策略会将请求集中在评分最低的实例上，此时盲目分发（blind spreading）的效果反而更好。通过使用校准后的成本，学习型路由器仅需六块 GPU 即可达到轮询策略使用七块 GPU 时的有效吞吐量。