AgentMemBench: A Systematic Benchmark for Evaluating Long-Term Memory Management Strategies in Conversational AI Agents

AgentMemBench：用于评估对话式 AI 智能体长期记忆管理策略的系统性基准测试

Long-term memory remains a critical bottleneck for conversational AI agents, whose finite context windows cannot support coherent recall across thousands of turns. We present AgentMemBench, a unified, reproducible benchmark evaluating five memory management strategies under identical conditions: in-context windowing (ICW), external key-value store (EKV), graph-based episodic memory (GEM), compression-based summarisation (CBS), and web-augmented memory (WAM).

长期记忆仍然是对话式 AI 智能体的关键瓶颈，其有限的上下文窗口无法支持跨越数千轮对话的连贯回溯。我们提出了 AgentMemBench，这是一个统一且可复现的基准测试，旨在相同条件下评估五种记忆管理策略：上下文窗口（ICW）、外部键值存储（EKV）、基于图的情景记忆（GEM）、基于压缩的摘要（CBS）以及网络增强记忆（WAM）。

All are assessed across three public datasets covering long-term multi-session dialogue (LoCoMo), task-oriented document grounding (MultiDoc2Dial), and persona-grounded multi-session chat (MSC), using Recall@k, MRR, nDCG@k, Answer F1, an LLM-judge Faithfulness score, Memory Footprint, and Latency over 491 annotated question turns. Generation and judging both use Qwen2.5-7B-Instruct (4-bit), with greedy decoding for determinism.

所有策略均在三个公共数据集上进行评估，涵盖长期多轮对话（LoCoMo）、任务导向的文档基础（MultiDoc2Dial）以及基于角色的多轮聊天（MSC）。评估指标包括 Recall@k、MRR、nDCG@k、答案 F1 分数、LLM 判定的忠实度（Faithfulness）分数、内存占用以及 491 个标注问题轮次中的延迟。生成和判定均使用 Qwen2.5-7B-Instruct (4-bit) 模型，并采用贪婪解码以确保确定性。

Our results show that (1) EKV dominates on every quality axis (macro Recall@5 0.792, MRR 0.677, F1 0.156, Faithfulness 0.354); (2) long-range recall is decisive: on LoCoMo, where the gold turn lies many sessions back, ICW, WAM, GEM, and CBS retrieve almost nothing (Recall@5 <= 0.005) while EKV alone reaches 0.573, showing that recency windows, summaries, and entity graphs collapse at long horizons and only dense retrieval scales; (3) CBS is the runner-up on retrieval (0.556); (4) WAM equals ICW on in-corpus recall by construction, since external results carry no in-corpus provenance; and (5) EKV’s recall advantage carries a footprint cost (~5,100 vs ~300 tokens for ICW/WAM), an explicit accuracy-efficiency trade-off.

研究结果表明：(1) EKV 在所有质量维度上均占据主导地位（宏观 Recall@5 为 0.792，MRR 为 0.677，F1 为 0.156，忠实度为 0.354）；(2) 长距离回溯至关重要：在 LoCoMo 数据集中，当目标轮次位于多个会话之前时，ICW、WAM、GEM 和 CBS 几乎无法检索到任何内容（Recall@5 <= 0.005），而唯有 EKV 达到了 0.573，这表明近因窗口、摘要和实体图在长跨度下会失效，只有密集检索具备扩展性；(3) CBS 在检索方面位居第二（0.556）；(4) WAM 在语料库内检索表现与 ICW 持平，因为外部结果不具备语料库内的溯源性；(5) EKV 的检索优势伴随着内存成本（约 5,100 tokens，而 ICW/WAM 仅约 300 tokens），这体现了准确性与效率之间的明确权衡。

We additionally evaluate two published memory systems (MemGPT/Letta, HippoRAG) against the same harness, and release all code, environment, and result artefacts for full reproducibility.

此外，我们还使用相同的测试框架评估了两个已发布的记忆系统（MemGPT/Letta 和 HippoRAG），并发布了所有代码、环境配置和结果工件，以实现完全的可复现性。