When Does Memory Help? A Cost-Aware Evaluation of Long-Term Memory in Tool-Using LLM Agents

记忆何时真正有用？工具使用型 LLM 智能体长期记忆的成本感知评估

Abstract: Long-term memory for LLM agents is evaluated today by conversational recall benchmarks (LoCoMo, LongMemEval), which measure question answering over dialogue history, not whether remembered facts change what a tool-using agent does.

摘要： 目前，大语言模型（LLM）智能体的长期记忆评估主要依赖于对话回溯基准（如 LoCoMo、LongMemEval）。这些基准衡量的是基于对话历史的问答能力，而非评估所记忆的事实是否真正改变了工具使用型智能体的行为。

We present MERIT (Memory Evaluation for Realistic Instrumented Tasks), a benchmark and harness that measures the marginal utility of memory for task-executing agents under explicit cost accounting. MERIT provides episodic tool-use tasks in three domains whose dependence on earlier-episode facts is verified by an automated leak check; a difficulty ladder ending in updated-fact recall; controlled memory corruption; and full token and dollar metering of every memory operation.

我们提出了 MERIT（现实工具化任务的记忆评估），这是一个基准测试与评估框架，旨在通过明确的成本核算来衡量记忆对执行任务的智能体的边际效用。MERIT 在三个领域提供了情境化工具使用任务，并通过自动化泄露检查验证了任务对早期情境事实的依赖性；它包含一个以更新事实回溯为终点的难度阶梯、受控的记忆损坏测试，以及对每一项记忆操作的完整 Token 和美元成本计量。

Across 23,440 scored episodes ($42.57), a two-generation pilot on gpt-4.1-mini and a preregistered 3-model x 3-seed grid (GPT-4.1, Claude Haiku 4.5; memory side held fixed), memory lifts dependent-task success from a leak-verified floor of 0.00 to 0.55-1.00.

在 23,440 个评分情境（总成本 42.57 美元）中，通过对 gpt-4.1-mini 进行的两代试点，以及预注册的“3 模型 x 3 随机种子”网格测试（GPT-4.1、Claude Haiku 4.5；记忆端保持固定），结果显示：记忆将依赖型任务的成功率从经泄露验证的基准线 0.00 提升到了 0.55-1.00。

On updated facts, embedding retrieval collapses unpredictably (0.30-0.95 across models; max seed gap 0.45), and agents act on a correctly retrieved value only 55% of the time, while update-on-write stores (a structured fact store and, notably, LLM summarization) remain at 0.70-1.00; the hybrid is worse than the fact store alone. A latest-generation spot-check (Claude Sonnet 5, gated on a clean full-replay control) reproduces the pattern.

在处理更新后的事实时，嵌入检索（embedding retrieval）的表现出现不可预测的崩溃（各模型间为 0.30-0.95；最大种子间隙为 0.45），且智能体仅在 55% 的情况下会根据正确检索到的值采取行动；相比之下，“写入时更新”存储（结构化事实存储，尤其是 LLM 摘要）保持在 0.70-1.00 的水平；混合模式的效果反而不如单一的事实存储。最新一代的抽查（Claude Sonnet 5，基于纯净的全重放控制）重现了这一模式。

Swapping a memory’s implementation moves task success by up to 60 points, and full replay is never economical: the best condition per domain delivers 2.7-3.9x its marginal utility per dollar. We release the benchmark, harness, and all traces.

更换记忆实现方式可使任务成功率波动高达 60 个百分点，且全量重放（full replay）在经济上从不可行：每个领域的最优条件所提供的边际效用是其成本的 2.7-3.9 倍。我们现已发布该基准测试、评估框架及所有追踪数据。