Freeze the Decoder, Heal the Encoder: Parameter-Efficient Adaptation for SVD-Based KV-Cache Compression

Comparing parameter-efficient fine-tuning recipes under a single, shared learning rate is a common but flawed practice: when the arms being compared have very different trainable-parameter counts, a shared rate can simultaneously depress the larger arms’ means and inflate their variance, manufacturing a large, seemingly multi-seed-significant advantage for the smallest arm that is not a real effect.

在单一共享学习率下比较参数高效微调方案是一种常见但存在缺陷的做法：当被比较的各组模型具有截然不同的可训练参数量时，共享学习率可能会同时抑制较大模型组的均值并放大其方差，从而为参数量最小的模型组制造出一种看似在多次随机种子下显著的优势，而这种优势并非真实存在。

We document this confound in a concrete setting: post-hoc SVD-based KV-cache compression, where an already-pretrained model is converted to a low-rank (multi-head-latent-attention-style) cache by factorizing its key/value weights into a down-projection (“encoder”) and an up-projection (“decoder”), after which a short fine-tune (“healing”) recovers the accuracy lost to truncation.

我们在一个具体的场景中记录了这种混淆因素：事后基于 SVD 的 KV 缓存压缩。在该场景中，通过将预训练模型的键/值权重分解为降维投影（“编码器”）和升维投影（“解码器”），将模型转换为低秩（多头潜在注意力风格）缓存，随后进行短时间的微调（“修复”）以恢复因截断而损失的精度。

Under a shared learning rate, freezing the decoder and healing only the encoder looks like a clear win over healing the decoder or both factors; once every arm is given its own tuned learning rate, that apparent advantage disappears, and encoder-only healing instead reaches parity with the alternatives, at a real, measured saving of 3x fewer trainable parameters and 3x less optimizer-state memory.

在共享学习率下，冻结解码器并仅修复编码器看起来明显优于修复解码器或同时修复两者；一旦为每一组模型分配了各自调整后的学习率，这种表面的优势便消失了，仅修复编码器的方法反而与替代方案达到了同等水平，并实现了 3 倍可训练参数减少和 3 倍优化器状态内存节省的实际测量收益。

We verify this parity with per-arm learning-rate tuning and three seeds per configuration on a vision-language model (Qwen2.5-VL-3B-Instruct), at the one compression ratio this protocol covers, and replicate it on a text-only testbed across two backbones.

我们通过对视觉语言模型（Qwen2.5-VL-3B-Instruct）进行分项学习率调整，并为每个配置设置三个随机种子，在协议涵盖的单一压缩比下验证了这种等效性，并在两个主干网络的纯文本测试平台上进行了复现。

Encoder-only healing is therefore a lower-memory drop-in recipe for retrofitting low-rank KV-cache compression at training time, and the shared-learning-rate pitfall we document and correct is a cautionary result for comparing any fine-tuning recipes whose arms differ in trainable-parameter count.

因此，仅修复编码器是一种在训练时改造低秩 KV 缓存压缩的低内存即插即用方案，而我们记录并纠正的共享学习率陷阱，对于比较任何可训练参数量存在差异的微调方案而言，是一个警示性的结论。