Where Should the KV Cache Live? Placement Policies Across GPU, CPU, and SSD for Long-Lived Sessions

KV Cache 该存放在哪里？针对长生命周期会话在 GPU、CPU 和 SSD 之间的放置策略

Abstract: GPU high bandwidth memory is scarce and expensive, and KV caches consume much of it as chats, agent loops, and document question answering accumulate state. Systems such as Mooncake, LMCache, FlexGen, InfiniGen, and AttentionStore extend GPU memory with CPU DRAM and SSD. The harder question is which blocks belong in each tier, when to move or evict them, and whether prefetching helps.

摘要： GPU 高带宽内存（HBM）既稀缺又昂贵，而随着聊天、智能体循环（agent loops）和文档问答任务不断积累状态，KV Cache 会消耗掉其中大部分内存。Mooncake、LMCache、FlexGen、InfiniGen 和 AttentionStore 等系统通过 CPU DRAM 和 SSD 来扩展 GPU 内存。更棘手的问题在于：哪些数据块应该存放在哪一层级，何时移动或驱逐它们，以及预取（prefetching）是否有帮助。

We study these choices in a discrete event simulator spanning GPU HBM, CPU DRAM, and SSD, calibrated against a random forest execution time predictor. We compare recency, reuse frequency, predicted reuse, and an EWMA predictor with prefetch lookahead across chat, agent, and document question answering workloads.

我们在一个涵盖 GPU HBM、CPU DRAM 和 SSD 的离散事件模拟器中研究了这些选择，并针对随机森林执行时间预测器进行了校准。我们对比了最近使用（recency）、重用频率（reuse frequency）、预测重用（predicted reuse）以及带有预取前瞻的 EWMA 预测器，并针对聊天、智能体和文档问答工作负载进行了测试。

Tiering supports 73.02 times more concurrent sessions per GPU and lowers cost per session by 62.04 times. These gains come from tier capacities of 1 plus 8 plus 64, not placement policy. Decode is compute bound at batch size one in our setup, so placement barely affects throughput. It mainly changes PCIe migration traffic and time to first token.

分层存储支持每个 GPU 的并发会话数增加 73.02 倍，并将每个会话的成本降低了 62.04 倍。这些收益来自于 1:8:64 的分层容量配置，而非放置策略本身。在我们的设置中，当批处理大小为一时，解码过程受限于计算能力，因此放置策略对吞吐量的影响微乎其微。它主要改变的是 PCIe 迁移流量和首字延迟（time to first token）。

Recency produces 2.30 times less migration traffic than reuse frequency for chat. Reuse frequency performs best for agents and document question answering. The existing predicted reuse policy is byte identical to recency, making its agent recommendation effectively recency. A genuine EWMA predictor changes behavior but still ranks behind reuse frequency on the workloads prediction was expected to help.

对于聊天任务，最近使用策略产生的迁移流量比重用频率策略少 2.30 倍。重用频率策略在智能体和文档问答任务中表现最佳。现有的预测重用策略在字节层面与最近使用策略完全相同，这使得其对智能体的推荐实际上等同于最近使用策略。真正的 EWMA 预测器虽然改变了行为，但在预期能够发挥作用的工作负载上，其表现仍落后于重用频率策略。

Prefetching does not justify its bandwidth cost. Across the policy and cache size grid, even an oracle with knowledge of future requests never beats no prefetch on migration traffic. Workload specific placement can reduce data movement, but the predicted reuse and prefetch recommendations are not supported as implemented.

预取机制无法抵消其带来的带宽成本。在各种策略和缓存大小的网格测试中，即使是拥有未来请求先验知识的“预言机”（oracle），在迁移流量方面也无法优于不使用预取的情况。针对特定工作负载的放置策略可以减少数据移动，但目前实现的预测重用和预取建议并未得到数据支持。