Disaggregation Is a Thousand-GPU Problem

解耦是千卡规模才有的问题

Every major inference framework shipped prefill-decode disaggregation this year. NVIDIA built it into Dynamo. SGLang made it the default for large-scale deployments. vLLM added a KV connector API to support it natively. The consensus is forming fast: split your prefill and decode onto separate GPU pools, and throughput improves. 今年，每一个主流推理框架都推出了预填充（prefill）与解码（decode）解耦功能。NVIDIA 将其内置于 Dynamo 中；SGLang 将其设为大规模部署的默认选项；vLLM 则增加了 KV 连接器 API 以实现原生支持。业界共识正在迅速形成：将预填充和解码任务拆分到不同的 GPU 池中，可以提高吞吐量。

The consensus is wrong for most teams. Doubleword’s analysis shows that a balanced disaggregated deployment matches colocated throughput, but at small GPU counts the rounding losses dominate: you can’t allocate fractional GPUs, so the specialization gains get eaten by incomplete worker utilization. The practical benefit at that scale is independent SLO tuning, not throughput. 但对于大多数团队来说，这个共识是错误的。Doubleword 的分析表明，一个均衡的解耦部署在吞吐量上与同置（colocated）部署相当，但在 GPU 数量较少时，取整带来的损失会占据主导地位：你无法分配零碎的 GPU，因此专业化带来的收益会被未充分利用的计算资源抵消。在这种规模下，其实际意义在于能够独立调整服务等级目标（SLO），而非提升吞吐量。

A June 2025 study evaluating hundreds of thousands of design points found that disaggregation is most effective for prefill-heavy traffic patterns and larger models. For the mixed-traffic workloads most teams actually run, queueing and inter-node KV cache transfer dominated end-to-end latency. The teams that disaggregated moved the bottleneck. They didn’t remove it. 2025 年 6 月的一项研究评估了数十万个设计点，发现解耦对于预填充密集型流量模式和大型模型最为有效。对于大多数团队实际运行的混合流量工作负载，排队和节点间的 KV 缓存传输才是端到端延迟的主要瓶颈。那些进行了解耦的团队只是转移了瓶颈，并没有消除它。

I ran into this on an inference workload serving a mid-size classification model. TPOT spiked under bursty traffic, and the first instinct was to separate prefill from decode. Instead, I enabled chunked prefill on the same GPU pool. TPOT stabilized. The problem was scheduling interference, and chunked prefill handled it without adding a network hop. 我在处理一个中型分类模型的推理任务时遇到了这个问题。在突发流量下，每个输出 token 的时间（TPOT）飙升，我的第一反应是将预填充与解码分离。但我转而尝试在同一个 GPU 池上启用了分块预填充（chunked prefill），TPOT 随即稳定了下来。问题的根源在于调度干扰，而分块预填充在无需增加网络跳转的情况下解决了这一问题。

What Disaggregation Solves: The Interference Problem

解耦解决了什么：干扰问题

Prefill and decode have opposite hardware profiles. Prefill is compute-bound: parallel matrix multiplications across the full input sequence, GPU compute utilization at 80 to 95 percent. Decode is memory-bandwidth-bound: sequential KV cache reads generating one token at a time, compute utilization below 5 percent on an H100. 预填充和解码具有截然不同的硬件特征。预填充是计算密集型的：它在整个输入序列上进行并行矩阵乘法，GPU 计算利用率可达 80% 到 95%。解码则是内存带宽密集型的：它通过顺序读取 KV 缓存一次生成一个 token，在 H100 上计算利用率低于 5%。

When both share a GPU, they fight over the same resources. A single large prefill request arriving mid-decode inflates time-per-output-token by 2 to 30x under bursty workloads. The decode batch stalls while prefill saturates the compute units. 当两者共享一个 GPU 时，它们会争夺相同的资源。在突发工作负载下，一个大型预填充请求在解码过程中到达，会导致每个输出 token 的时间（TPOT）增加 2 到 30 倍。解码批次会因为预填充占满计算单元而停滞。

DistServe proved the fix works at scale: separating prefill and decode onto dedicated pools served 7.4x more requests within the same latency constraints. At the scale DistServe benchmarked, disaggregation is unambiguously the right call. DistServe 证明了这种方案在大规模场景下的有效性：将预填充和解码分离到专用池中，在相同的延迟约束下，请求处理能力提升了 7.4 倍。在 DistServe 测试的规模下，解耦无疑是正确的选择。

Chunked Prefill: The Fix Most Teams Actually Need

分块预填充：大多数团队真正需要的解决方案

Chunked prefill breaks long prefill requests into smaller chunks and interleaves them with decode batches on the same GPU. No separate node pools. No KV cache transfer over the network. No P:D ratio to tune. 分块预填充将长预填充请求拆分为较小的块，并将其与同一 GPU 上的解码批次交错执行。无需独立的节点池，无需通过网络传输 KV 缓存，也无需调整预填充与解码（P:D）的比例。

TNG Technology Consulting measured a 50 percent increase in total token throughput using standard vLLM with chunked prefill enabled. The decode batches still ran between prefill chunks on the same hardware, but the scheduling interference dropped to a level most production workloads can tolerate. TNG Technology Consulting 的测试显示，在使用启用了分块预填充的标准 vLLM 时，总 token 吞吐量提升了 50%。虽然解码批次仍然在同一硬件上的预填充块之间运行，但调度干扰降低到了大多数生产工作负载可以容忍的水平。

Think of it like a toll plaza on a highway: instead of closing all lanes for a single oversized truck, chunked prefill lets the truck pass through one lane at a time while regular traffic keeps moving through the others. 可以把它想象成高速公路上的收费站：与其为了让一辆超大型卡车通过而关闭所有车道，分块预填充允许卡车一次通过一个车道，同时普通车辆可以继续在其他车道通行。

Three Costs of Disaggregation: What the Explainers Skip

解耦的三大代价：那些科普文章忽略的事

The KV transfer tax. Every request that finishes prefill must ship its KV cache to a decode node over the network. For a 70B-parameter model, that’s roughly 2.6 GB per request. KV 传输税。每个完成预填充的请求都必须通过网络将其 KV 缓存发送到解码节点。对于一个 70B 参数的模型，每个请求大约需要传输 2.6 GB 数据。

When prefill and decode share a node, the KV cache stays in GPU memory. Once you disaggregate across nodes, that transfer hits the network interconnect, and the available bandwidth drops by orders of magnitude depending on your topology. 当预填充和解码共享一个节点时，KV 缓存保留在 GPU 内存中。一旦跨节点解耦，传输就会经过网络互连，根据拓扑结构的不同，可用带宽可能会下降几个数量级。

The operational surface. Disaggregation doubles your infrastructure management. You now run separate prefill and decode node pools, each with its own scaling policy. The P:D ratio depends on your workload mix. 运维复杂度。解耦使你的基础设施管理工作量翻倍。你现在需要运行独立的预填充和解码节点池，每个池都有自己的扩展策略。P:D 比例取决于你的工作负载组合。

The silent failure cliff. At low concurrency, disaggregated serving works fine. At production concurrency, it breaks in ways that produce no errors. 静默失败的悬崖。在低并发下，解耦服务运行良好。但在生产环境的高并发下，它会以一种不产生错误的方式崩溃。

The Decision Math: Three Conditions That Must Hold

决策逻辑：必须满足的三个条件

Disaggregation pays for itself only when three conditions hold simultaneously: 只有同时满足以下三个条件，解耦才具有经济效益：

Enough GPUs for clean allocation. DeepSeek needed thousands of GPUs before the P:D ratio produced integer node counts that matched their traffic mix. At 8 to 16 GPUs, your ratio options are 1:7 or 2:6, which rarely align with your actual compute needs.
拥有足够的 GPU 以进行合理的分配。 DeepSeek 在拥有数千张 GPU 后，其 P:D 比例才能产生与流量组合匹配的整数节点数。在 8 到 16 张 GPU 的规模下，你的比例选项只有 1:7 或 2:6，这很难与你的实际计算需求相匹配。