AdaMem: Adaptive Memory Token Allocation for Soft Compression in Retrieval-Augmented Generation

AdaMem：检索增强生成中用于软压缩的自适应内存 Token 分配

Abstract: Retrieval-augmented generation (RAG) improves language models with retrieved evidence, but processing many long passages is costly and can introduce distracting information. Soft compression addresses this challenge by encoding passages as compact sequences of continuous memory embeddings before generation. However, existing methods typically assign each retained passage an identical number of memory embeddings, irrespective of its query-specific relevance.

摘要： 检索增强生成（RAG）通过检索到的证据改进了语言模型，但处理大量长篇段落不仅成本高昂，还可能引入干扰信息。软压缩通过在生成前将段落编码为连续内存嵌入的紧凑序列来应对这一挑战。然而，现有方法通常为每个保留的段落分配相同数量的内存嵌入，而不考虑其针对特定查询的相关性。

To address this, we propose AdaMem, a relevance-guided soft-compression framework that maps learned passage-relevance estimates to a query-dependent allocation of a fixed memory-token budget. A shared query-conditioned compressor produces both continuous passage memories and relevance scores in a single pass; a deterministic allocation rule assigns more memory tokens to higher-scoring passages and can omit low-scoring ones.

为了解决这一问题，我们提出了 AdaMem，这是一个相关性引导的软压缩框架，它将学习到的段落相关性估计映射到固定内存 Token 预算的查询依赖分配中。一个共享的查询条件压缩器可以在单次传递中同时生成连续的段落内存和相关性分数；确定性的分配规则会为高分段落分配更多的内存 Token，并可以忽略低分段落。

Across six open-domain QA benchmarks, AdaMem consistently outperforms OSCAR (the closely matched soft-compression baseline that uses uniform allocation) as well as other soft-compression methods at matched memory budgets. Under standard 16$\times$ compression, AdaMem improves sub-string match by up to 3.2 points (5.5%) over uniform allocation baseline, with an average relative gain of 3.4%; under aggressive 64$\times$ compression the average relative gain grows to 14.6%, with a maximum of 9.8 points (19.7%) on PopQA.

在六个开放域问答（QA）基准测试中，AdaMem 在相同的内存预算下，始终优于 OSCAR（使用均匀分配的紧密匹配软压缩基线）以及其他软压缩方法。在标准的 16 倍压缩下，AdaMem 的子字符串匹配度比均匀分配基线提高了 3.2 个百分点（5.5%），平均相对增益为 3.4%；在激进的 64 倍压缩下，平均相对增益增长至 14.6%，在 PopQA 上最高提升了 9.8 个百分点（19.7%）。

AdaMem matches the answer quality of the uncompressed at up to 4$\times$ lower inference latency than full context baseline. AdaMem retains an efficiency profile comparable to the uniform-compression baseline, while achieving up to 4$\times$ lower inference latency than full-context inference. Thus, relevance-guided memory allocation is particularly effective when retrieval pools are large and the available memory budget is tight.

AdaMem 在推理延迟比全上下文基线低 4 倍的情况下，达到了与未压缩模型相当的回答质量。AdaMem 保持了与均匀压缩基线相当的效率，同时实现了比全上下文推理低 4 倍的推理延迟。因此，当检索池较大且可用内存预算紧张时，相关性引导的内存分配尤为有效。