KVBoost: Chunk-Level Key-Value Cache Reuse with Deviation-Guided Recomputation for Efficient Large Language Model Inference

KVBoost: Chunk-Level Key-Value Cache Reuse with Deviation-Guided Recomputation for Efficient Large Language Model Inference

KVBoost：基于偏差引导重计算的块级键值缓存重用，实现高效大语言模型推理

Abstract: Transformer-based large language models (LLMs) incur high prefill latency because key-value (KV) tensors must be recomputed for each request. Existing prefix-caching systems reduce this cost but require prompts to share a leading contiguous prefix, limiting effectiveness when shared content appears at arbitrary positions. We present KVBoost, a chunk-level KV cache reuse system for HuggingFace-compatible decoder models that enables reuse regardless of content position.

摘要： 基于 Transformer 的大语言模型（LLM）会产生较高的预填充（prefill）延迟，因为键值（KV）张量必须为每个请求重新计算。现有的前缀缓存系统虽然降低了这一成本，但要求提示词（prompts）共享一个连续的前导前缀，这限制了当共享内容出现在任意位置时的有效性。我们提出了 KVBoost，这是一个针对兼容 HuggingFace 的解码器模型的块级 KV 缓存重用系统，无论内容处于什么位置，都能实现缓存重用。

KVBoost introduces a dual-hash keying scheme that separates positional identity (prefix hash) from content identity (content hash), supporting both exact and approximate cache matches. To address attention boundary errors from independently cached chunks, KVBoost employs two repair strategies: SelectiveRecompute, which re-encodes boundary regions, and CacheBlendRecompute, which identifies and recomputes high-deviation tokens after a probe pass.

KVBoost 引入了一种双哈希键控方案，将位置标识（前缀哈希）与内容标识（内容哈希）分离开来，从而支持精确和近似的缓存匹配。为了解决独立缓存块带来的注意力边界错误，KVBoost 采用了两种修复策略：SelectiveRecompute（选择性重计算），用于重新编码边界区域；以及 CacheBlendRecompute（缓存融合重计算），用于在探测通过后识别并重新计算高偏差标记（tokens）。

The system further incorporates asymmetric KV quantization (int8/int4), adaptive chunk boundary splitting, and importance-weighted eviction under a fixed memory budget. Evaluated on Qwen/Qwen2.5-3B over 1,000 bug-localization samples, KVBoost achieves a 4.49x reduction in time-to-first-token (142.4 ms vs. 639.1 ms) and outperforms prefix caching by 16%, with no loss in accuracy (99.2% vs. 99.1%). KVBoost provides a practical, memory-bounded inference acceleration layer compatible with RoPE-based models without architectural modification.

该系统还结合了非对称 KV 量化（int8/int4）、自适应块边界分割以及在固定内存预算下的重要性加权驱逐机制。在 Qwen/Qwen2.5-3B 模型上针对 1,000 个错误定位样本的评估显示，KVBoost 将首字延迟（time-to-first-token）降低了 4.49 倍（142.4 毫秒对比 639.1 毫秒），性能比前缀缓存提升了 16%，且精度没有损失（99.2% 对比 99.1%）。KVBoost 提供了一种实用的、内存受限的推理加速层，无需修改架构即可兼容基于 RoPE 的模型。