Real Long-Term Memory for AI: A 50-Million-Token Window That Is Faster and Cheaper Than Recompute

Computer Science > Computation and Language arXiv:2610.10845 (cs) [Submitted on 7 Oct 2026] Title: Real Long-Term Memory for AI: A 50-Million-Token Window That Is Faster and Cheaper Than Recompute Authors: Sietse Schelpe.

计算机科学 > 计算与语言 arXiv:2610.10845 (cs) [提交于 2026 年 10 月 7 日] 标题：人工智能的真正长期记忆：一个比重新计算更快、更便宜的 5000 万 Token 窗口 作者：Sietse Schelpe。

Abstract: A large language model can only use the text that fits in its context window, and it recomputes its internal key-value (KV) state for a prompt every time the prompt is sent. We test a memory layer, the public package galahad-kv, that saves the KV state of each block of about 16,000 tokens to encrypted local NVMe disk and loads it back later, byte-exact, without recomputing it.

摘要：大型语言模型只能使用适合其上下文窗口的文本，并且每次发送提示词时，它都会重新计算其内部的键值（KV）状态。我们测试了一个名为 galahad-kv 的公共软件包作为内存层，它将大约 16,000 个 Token 的每个数据块的 KV 状态保存到加密的本地 NVMe 磁盘中，并在稍后以字节精确的方式加载回来，而无需重新计算。

We ran it on 50,000,000 tokens of real public text, served through vLLM on one NVIDIA H100, with Gemma 4 12B and Gemma 4 31B. Every block we probed was loaded back from the encrypted store with no recompute (100 of 100, at depths from 0 to 50M tokens) on both models.

我们在 5000 万个真实公共文本 Token 上运行了该系统，通过单张 NVIDIA H100 上的 vLLM 提供服务，并使用了 Gemma 4 12B 和 Gemma 4 31B 模型。在两种模型上，我们探测的每一个数据块都从加密存储中加载回来，且无需重新计算（在 0 到 5000 万 Token 的深度范围内，100 次测试全部成功）。

Loading a block was 2.8x to 4.3x faster than recomputing it and used 8.8x to 12.3x less GPU energy, and GPU memory stayed flat over the whole 50M-token stream. Asked about facts planted millions of tokens earlier, the 12B model gave the right answer 82 times out of 100 and the 31B model 98 times out of 100. Neither model made up an answer.

加载一个数据块的速度比重新计算快 2.8 到 4.3 倍，GPU 能耗降低了 8.8 到 12.3 倍，且在整个 5000 万 Token 的处理流中，GPU 内存占用保持平稳。当被问及数百万 Token 之前植入的事实时，12B 模型在 100 次测试中答对了 82 次，31B 模型答对了 98 次。两个模型都没有编造答案。

The limits are as follows. This is reuse of stored state, not a wider attention window: one block is loaded at a time, and how well a question is answered depends on the model. Writing the memory is a one-time cost, and the store takes terabytes of local NVMe disk. We describe the test protocol, which is built to resist common ways of gaming long-context benchmarks, and give a single-GPU reproduction that uses public software and a free licence for the package.

其局限性如下：这是对已存储状态的重用，而非更宽的注意力窗口：一次只能加载一个数据块，且问题的回答质量取决于模型本身。写入内存是一次性成本，且存储需要占用数 TB 的本地 NVMe 磁盘空间。我们描述了测试协议，该协议旨在抵御常见的长上下文基准测试作弊方式，并提供了一个使用公共软件和该软件包免费许可的单 GPU 复现方案。