VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs

VarRate：面向长上下文大语言模型的无需训练、可变速率 KV Cache 压缩技术

Abstract: The key-value (KV) cache is the main memory bottleneck in long-context large language model (LLM) inference. Two leading training-free families are both structurally limited: token-selection methods (SnapKV, Ada-KV) score importance from an observation window and evict low-scoring tokens, but eviction is irreversible — so when the importance signal degrades under query-agnostic reuse, accuracy collapses by 11-15 points; uniform low-rank coding keeps every token but spends equal rank everywhere, wasting budget.

摘要： 键值（KV）缓存是长上下文大语言模型（LLM）推理中的主要内存瓶颈。目前两种主流的无需训练压缩方法在结构上都存在局限性：基于 Token 选择的方法（如 SnapKV、Ada-KV）通过观察窗口对重要性进行评分并剔除低分 Token，但这种剔除是不可逆的——因此，当重要性信号在与查询无关的重用场景下减弱时，模型精度会下降 11-15 个百分点；而统一低秩编码方法虽然保留了所有 Token，却在所有位置分配相同的秩，导致了预算浪费。

We observe that both failures share one cure: rank should be allocated, not evicted. We present VarRate, a training-free KV codec that assigns each token a variable low-rank budget by its query salience, keeping every token at a nonzero rank. Comparable adaptive-rank codecs reach this allocation only through training; VarRate requires none.

我们观察到，这两种缺陷可以通过同一个方案解决：应该通过分配秩（rank）来处理，而不是剔除 Token。我们提出了 VarRate，这是一种无需训练的 KV 编解码器，它根据查询显著性为每个 Token 分配可变的低秩预算，确保每个 Token 都保持非零秩。同类的自适应秩编解码器通常需要通过训练才能实现这种分配，而 VarRate 完全不需要训练。

Because no token is dropped, it degrades by only 3.5-5.5 points where query-aware selection collapses. At a matched 20% budget on LongBench (16 tasks), VarRate stays within 0.8 points of the uncompressed model on both Llama-3.1-8B and Qwen2.5-7B. Averaged over the two, it is the strongest matched-memory compressor. It significantly beats its uniform-rank ablation on both models. Against KVzip, a method purpose-built for query-agnostic reuse, it is accuracy-equivalent in three of four settings and within a point overall, at about one-eighth the prefill overhead.

由于没有丢弃任何 Token，在查询感知选择方法失效的情况下，VarRate 的精度仅下降 3.5-5.5 个百分点。在 LongBench（16 项任务）的 20% 预算限制下，VarRate 在 Llama-3.1-8B 和 Qwen2.5-7B 模型上的表现与未压缩模型相比，精度差距均在 0.8 个百分点以内。综合来看，它是目前同等内存占用下性能最强的压缩器。在两个模型上，它都显著优于其统一秩的消融版本。与专门为查询无关重用而设计的 KVzip 相比，它在四分之三的设置中精度相当，整体差距在 1 个百分点以内，且预填充（prefill）开销仅为后者的八分之一左右。