MemTrace: Probing What Final Accuracy Misses in Long-Term Memory

MemTrace：探究长期记忆中被最终准确率所忽略的内容

LLM agents increasingly maintain long-term memory of user facts across sessions. Yet such memory is usually evaluated by aggregating accuracy over question rows or episodes. Because this approach scores question rows independently, even when several questions probe the same fact, it cannot show how that fact behaves as conditions change.

大语言模型（LLM）智能体在跨会话过程中，越来越多地需要维护关于用户事实的长期记忆。然而，此类记忆通常是通过汇总问题行或片段的准确率来进行评估的。由于这种方法是独立地对每个问题行进行评分，即使多个问题针对的是同一个事实，它也无法展示该事实在条件变化时的表现。

We introduce MemTrace, a benchmark whose unit of measurement is the knowledge point: a single typed fact about the user, rather than an individual question. MemTrace probes each fact along three controlled dimensions: memory age, defined by how many sessions ago the fact appeared in the history; question type, covering current state, earlier state, and trajectory of change; and evidence condition, covering present, missing, and contradicted-by-false-premise settings.

我们引入了 MemTrace，这是一个以“知识点”为度量单位的基准测试：即关于用户的单个特定事实，而非单个问题。MemTrace 从三个受控维度对每个事实进行探测：记忆时效（定义为该事实在历史记录中出现在多少个会话之前）；问题类型（涵盖当前状态、先前状态以及变化轨迹）；以及证据条件（涵盖证据存在、缺失以及被错误前提反驳的情境）。

Evaluating 13 memory-system configurations across four paradigms, we find that similar pooled accuracy hides different failures: recovering a fact’s current and earlier states does not imply tracking how it changed, and safe abstention does not imply correcting a false premise. The dominant bottleneck is evidence use, not retrieval: when systems fail, the evidence was retrievable 10 times more often than it was missing. These results suggest that improving long-term memory requires better use of reachable evidence, not simply more storage or retrieval.

通过对四种范式下的 13 种记忆系统配置进行评估，我们发现相似的汇总准确率掩盖了不同的失效模式：能够恢复事实的当前和先前状态，并不意味着能够追踪其变化过程；而安全的“弃权”回答也不意味着能够纠正错误的前提。主要的瓶颈在于证据的使用，而非检索：当系统失败时，证据可被检索到的频率是证据缺失频率的 10 倍。这些结果表明，提升长期记忆能力的关键在于更好地利用可获取的证据，而非仅仅增加存储或检索量。