Linear-Time Context Compression with State-Space Encoding
Computer Science > Computation and Language arXiv:2610.33006 (cs) [Submitted on 7 Oct 2026] Title:Linear-Time Context Compression with State-Space Encoding Authors:Marc Serra Ortega, Andrey Barsky, Carlos Boned Riera, Tomás Ockier, Dimosthenis Karatzas View a PDF of the paper titled Linear-Time Context Compression with State-Space Encoding, by Marc Serra Ortega and 4 other authors View PDF HTML (experimental)
计算机科学 > 计算与语言 arXiv:2610.33006 (cs) [提交于 2026 年 10 月 7 日] 标题：基于状态空间编码的线性时间上下文压缩 作者：Marc Serra Ortega, Andrey Barsky, Carlos Boned Riera, Tomás Ockier, Dimosthenis Karatzas 查看标题为《基于状态空间编码的线性时间上下文压缩》的论文 PDF，作者为 Marc Serra Ortega 及其他 4 位作者 查看 PDF HTML（实验性）
Abstract:Transformer-based large language models (LLMs) impose substantial computation and memory costs as context sizes increase, motivating methods that compress a context into a shorter representation before inference. Existing soft-prompt compression methods typically use the same Transformer backbone as used in the target model, where producing the compressed representation leads to significant latency overheads. This restricts them to the offline setting where context is compressed once and then reused across many queries.
摘要：随着上下文长度的增加，基于 Transformer 的大语言模型（LLM）会带来巨大的计算和内存成本，这促使人们研究在推理前将上下文压缩为更短表示的方法。现有的软提示（soft-prompt）压缩方法通常使用与目标模型相同的 Transformer 主干网络，而生成压缩表示的过程会导致显著的延迟开销。这限制了它们仅能用于离线场景，即上下文被压缩一次后在多次查询中重复使用。
We propose the Boa Compressor, a plug-and-play online context compressor built on a Mamba state-space model (SSM) that encodes arbitrary-length context in a single linear-time pass, producing a representation on which we condition a frozen target LLM. The SSM is trained by distillation, matching the next-token logit distribution from the LLM ingesting the full context (as teacher) to the same LLM ingesting the compressed representation (as the student).
我们提出了 Boa Compressor，这是一种基于 Mamba 状态空间模型（SSM）的即插即用型在线上下文压缩器。它能在单次线性时间遍历中对任意长度的上下文进行编码，生成一个用于调节冻结目标 LLM 的表示。该 SSM 通过蒸馏进行训练，使输入完整上下文的 LLM（作为教师）的下一个 token 的 Logit 分布与输入压缩表示的同一 LLM（作为学生）的分布相匹配。
At equal compression ratios, Boa matches or surpasses the performance of existing state-of-the-art offline compression baselines, while being an order of magnitude faster than Transformer-based implementations, thereby reducing end-to-end latency rather than adding compression time overhead. We demonstrate that Boa also scales well to longer contexts and to compression ratios as aggressive as 16x. By enabling effective compression that is cheap enough to run on every query at inference time, Boa becomes a practical and general tool for efficient long-context inference.
在相同的压缩比下，Boa 的性能达到或超过了现有的最先进离线压缩基准，同时其速度比基于 Transformer 的实现快一个数量级，从而降低了端到端延迟，而不是增加压缩时间开销。我们证明了 Boa 在处理更长上下文和高达 16 倍的激进压缩比时也具有良好的扩展性。通过实现足够廉价、可在推理时对每次查询运行的有效压缩，Boa 成为了一种用于高效长上下文推理的实用且通用的工具。