MoEZip: Routing-Aware KV Cache Compression for Sparse Mixture-of-Experts LLMs

查看原文 / View Original 编辑 / Edit

Computer Science > Machine Learning arXiv:2610.30206 (cs) [Submitted on 21 Oct 2026] Title: MoEZip: Routing-Aware KV Cache Compression for Sparse Mixture-of-Experts LLMs Authors: Minsang Kim, Seungjun Baek.

计算机科学 > 机器学习 arXiv:2610.30206 (cs) [提交于 2026 年 10 月 21 日] 标题：MoEZip：针对稀疏混合专家大语言模型的路由感知 KV 缓存压缩作者：Minsang Kim, Seungjun Baek。

Abstract: Mixture-of-Experts (MoE) language models scale model capacity efficiently by activating a small subset of parameters per token. However, in long-context inference, memory remains a major bottleneck because the KV cache grows linearly with sequence length.

摘要：混合专家（MoE）语言模型通过在每个 token 中激活一小部分参数来有效地扩展模型容量。然而，在长上下文推理中，内存仍然是一个主要瓶颈，因为 KV 缓存随序列长度线性增长。

Existing KV cache compression methods reduce this cost for dense transformers, but overlook a problem specific to MoE: KV eviction can change which experts are selected during answer generation, degrading the generation quality.

现有的 KV 缓存压缩方法降低了密集型 Transformer 的这种成本，但忽略了 MoE 特有的一个问题：KV 驱逐可能会改变回答生成过程中专家选择的结果，从而降低生成质量。

We propose MoEZip, a routing-aware KV cache compression for sparse MoE language models. To estimate the sensitivity of expert routing to perturbation, we propose a novel metric based on the Fisher information matrix of routing distributions.

我们提出了 MoEZip，这是一种针对稀疏 MoE 语言模型的路由感知 KV 缓存压缩方法。为了评估专家路由对扰动的敏感性，我们提出了一种基于路由分布的 Fisher 信息矩阵的新型度量标准。

MoEZip combines this metric with attention weights for routing-aware scoring of contexts. Finally, we compose a complementary scoring system that combines eviction methods for dense and sparse-MoE architectures.

MoEZip 将该度量标准与注意力权重相结合，用于上下文的路由感知评分。最后，我们构建了一个互补的评分系统，结合了针对密集型和稀疏 MoE 架构的驱逐方法。

Experiments show that MoEZip achieves robust performance under aggressive cache compression on various benchmarks. At a KV retention ratio of 10%, MoEZip improves over the SoTA baseline by +24 and +20 points on average on LongBench and RULER, corresponding to 2.3× and 4.9× higher scores, respectively.

实验表明，MoEZip 在各种基准测试的激进缓存压缩下均实现了稳健的性能。在 10% 的 KV 保留率下，MoEZip 在 LongBench 和 RULER 上平均比当前最优（SoTA）基准分别提高了 +24 和 +20 分，相当于得分分别提升了 2.3 倍和 4.9 倍。