Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving

针对长上下文服务中 KV-Cache 优化的任务质量与系统性能基准测试

Large language model serving is increasingly limited by KV-cache growth under long-context workloads, yet existing KV-cache compression techniques are difficult to compare because they were evaluated on different models, tasks, budgets, and serving stacks. 在大语言模型服务中，长上下文负载下的 KV-Cache 增长日益成为瓶颈。然而，现有的 KV-Cache 压缩技术由于在不同的模型、任务、预算和推理架构上进行评估，导致它们之间难以进行横向比较。

This paper presents a workload-aware benchmark of representative KV-cache optimization mechanisms spanning quantization, pruning, and merging, including KIVI, TurboQuant, SnapKV, and CaM, evaluated on LongBench-style multi-document QA, single-document QA, few-shot learning, and summarization workloads using Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.3. 本文提出了一个面向负载的基准测试，涵盖了量化、剪枝和合并等代表性 KV-Cache 优化机制（包括 KIVI、TurboQuant、SnapKV 和 CaM）。研究使用 Llama-3.1-8B-Instruct 和 Mistral-7B-Instruct-v0.3 模型，在 LongBench 风格的多文档问答、单文档问答、少样本学习和摘要生成等负载上进行了评估。

The benchmark measures task quality, mean output throughput, mean time-to-first-token, and realized compression ratio across context-length buckets. 该基准测试衡量了不同上下文长度区间内的任务质量、平均输出吞吐量、平均首字延迟（Time-to-First-Token）以及实际压缩比。

The results show that the compression ratio alone is a poor predictor of end-to-end performance. KIVI4 provides the most stable quality across models, SnapKV delivers the strongest long-context throughput, and CaM yields large gains on selected QA workloads but exhibits substantial workload sensitivity in both quality and realized compression ratio. 结果表明，仅凭压缩比无法准确预测端到端的性能表现。KIVI4 在不同模型间提供了最稳定的质量；SnapKV 实现了最强的长上下文吞吐量；而 CaM 在特定问答负载下表现优异，但在质量和实际压缩比方面表现出显著的负载敏感性。

These findings motivate workload-aware selection of KV-cache mechanisms rather than one-size-fits-all compression and provide deployment guidance for long-context serving systems. 这些发现表明，针对 KV-Cache 机制的选择应基于负载特性，而非采用“一刀切”的压缩方案，并为长上下文服务系统的部署提供了指导建议。