KVFetch: Temporal Prefetching for the Missing Half of KV Cache Compression

查看原文 / View Original 编辑 / Edit

Computer Science > Machine Learning arXiv:2610.08811 (cs) [Submitted on 23 Sep 2026] Title:KVFetch: Temporal Prefetching for the Missing Half of KV Cache Compression Authors:Linfeng Dong View a PDF of the paper titled KVFetch: Temporal Prefetching for the Missing Half of KV Cache Compression, by Linfeng Dong View PDF HTML (experimental)

计算机科学 > 机器学习 arXiv:2610.08811 (cs) [提交于 2026 年 9 月 23 日] 标题：KVFetch：针对 KV 缓存压缩缺失部分的临时预取作者：Linfeng Dong 查看标题为《KVFetch：针对 KV 缓存压缩缺失部分的临时预取》的论文 PDF，作者：Linfeng Dong 查看 PDF HTML（实验性）

Abstract:As context windows scale to tens or hundreds of thousands of tokens, KV cache compression has become essential for efficient LLM inference. Existing methods fall into three families: score-based eviction, summary compensation, and offload-and-recall. Yet all three decide what to keep or recall by content relevance to the current query. We show this shared design is structurally incomplete.

摘要：随着上下文窗口扩展到数万或数十万个 token，KV 缓存压缩对于高效的大语言模型（LLM）推理变得至关重要。现有方法分为三大类：基于分数的驱逐、摘要补偿以及卸载与召回。然而，这三种方法都是根据内容与当前查询的相关性来决定保留或召回哪些内容。我们证明这种共同的设计在结构上是不完整的。

A cache supports two access modes: associative lookup by content and sequential traversal by position; current compressors implement only the first. The gap matters in practice: retrieval-augmented generation, code completion, and structured-data extraction all require the model to reproduce identifiers, field values, or code tokens verbatim from the context.

缓存支持两种访问模式：基于内容的关联查找和基于位置的顺序遍历；当前的压缩器仅实现了前者。这种差距在实践中很重要：检索增强生成、代码补全和结构化数据提取都要求模型从上下文中逐字重现标识符、字段值或代码 token。

Under compression, content-based eviction retains the head of such a sequence but discards its continuation, causing verbatim copying to break irreversibly midway, a failure we call sequential forgetting. This failure resists better scoring, larger budgets, summary compensation, and dynamic re-scoring; it is the dominant source of remaining quality loss under compression.

在压缩下，基于内容的驱逐会保留此类序列的头部，但会丢弃其后续部分，导致逐字复制在中途不可逆地中断，我们将这种失败称为“顺序遗忘”。这种失败无法通过更好的评分、更大的预算、摘要补偿和动态重评分来解决；它是压缩下剩余质量损失的主要来源。

We propose KVFetch, a training-free, drop-in framework that opens a temporal recall channel for any score-based compressor. It demotes evicted candidates to a quantized cold tier, detects active copying through a monotone read pointer, and prefetches positional successors into fixed-size hot-tier slots without increasing attention cost.

我们提出了 KVFetch，这是一个无需训练、可直接使用的框架，它为任何基于分数的压缩器开启了一个临时召回通道。它将驱逐的候选对象降级到量化的冷层，通过单调读取指针检测主动复制，并将位置后续项预取到固定大小的热层槽中，而不会增加注意力成本。

On RULER-16K under an iso-budget control, KVFetch recovers verbatim copying from 0.8 to 78.4 and raises the 13-task average by +8.4, with gains concentrating on tasks that require sequential access. On LongBench, where no task requires sequential access, the channel remains dormant and imposes no cost.

在等预算控制下的 RULER-16K 测试中，KVFetch 将逐字复制能力从 0.8 恢复到 78.4，并将 13 项任务的平均得分提高了 +8.4，收益集中在需要顺序访问的任务上。在不需要顺序访问任务的 LongBench 上，该通道保持休眠状态且不产生任何成本。