MemArena: An Ego-Centric Benchmark for On-Device Agentic Personal Memory Assistants at Scale

MemArena：面向端侧智能个人记忆助手的以自我为中心的基准测试

Abstract: Edge-deployed personal memory assistants must handle private interpersonal conversations on-device with open-weight models. Yet, existing memory benchmarks often under-test the combination of activity-dense interaction, ego-centric perspective, and coherent multi-session worlds. MemArena fills these gaps with a single-world conversational benchmark built with its MASim agent simulator, for 50 agents over 15 days (10.3M dialog-text tokens, 24.1K text-only ego-observed tokens/agent/day). With the interaction history, it co-generates ground truth over six recall, reasoning, and trustworthiness evaluation dimensions.

摘要： 部署在边缘侧的个人记忆助手必须使用开放权重模型在设备端处理私密的个人对话。然而，现有的记忆基准测试往往未能充分评估高频交互、以自我为中心的视角以及连贯的多会话世界之间的结合。MemArena 通过其 MASim 智能体模拟器构建了一个单一世界的对话基准测试，填补了这些空白。该测试涵盖了 50 个智能体在 15 天内的交互（共 1030 万个对话文本 Token，每个智能体每天产生 2.41 万个仅文本的自我观察 Token）。利用交互历史，该基准在记忆召回、推理和可信度等六个评估维度上共同生成了基准真值（Ground Truth）。

We evaluate five open-weight readers with Vanilla context, BM25-RAG, Oracle retrieval, Memobase, and MemSearch as memory backends. Three results stand out: (1) Memory-backend choice matters more for content accuracy: At Qwen3-0.6B, Memobase-to-MemSearch gains +32.5/+19.2 pp, exceeding MemSearch reader scaling (+10.6/+6.8 pp). (2) Permission-aware access fails universally, with Oracle leaking heavily and other backends too timid to disclose. (3) Search latency bites only at very small reader: on a Spark GB10 edge node, memory-search adds a moderate and fixed 87/7/48 ms (BM25-RAG/Memobase/MemSearch) that composes a small part of TTFT for most reader-backend combinations.

我们评估了五种开放权重阅读器，并分别使用 Vanilla 上下文、BM25-RAG、Oracle 检索、Memobase 和 MemSearch 作为记忆后端。研究得出三个显著结论：（1）记忆后端的选择对内容准确性的影响更大：在 Qwen3-0.6B 模型下，从 Memobase 切换到 MemSearch 带来了 +32.5/+19.2 个百分点的提升，超过了仅通过扩展 MemSearch 阅读器规模带来的提升（+10.6/+6.8 个百分点）。（2）权限感知访问在所有测试中均表现不佳，Oracle 存在严重泄露，而其他后端则过于保守，不敢披露信息。（3）搜索延迟仅在极小规模的阅读器上表现明显：在 Spark GB10 边缘节点上，内存搜索增加了适度且固定的 87/7/48 毫秒（分别对应 BM25-RAG/Memobase/MemSearch）的延迟，这在大多数阅读器与后端的组合中仅占首字生成时间（TTFT）的一小部分。

Code, the MASim simulator, and the MemArena-L benchmark will be released upon acceptance.

代码、MASim 模拟器以及 MemArena-L 基准测试将在论文被录用后发布。