SRRM: A Retrieval-Reconstruction Benchmark for Evaluating Long-Context Memory in Small Language Models

查看原文 / View Original 编辑 / Edit

Computer Science > Computation and Language arXiv:2611.05386 (cs) [Submitted on 4 Aug 2026] Title: SRRM: A Retrieval-Reconstruction Benchmark for Evaluating Long-Context Memory in Small Language Models Authors: Christopher Alberto Hamonangan.

计算机科学 > 计算与语言 arXiv:2611.05386 (cs) [提交于 2026 年 8 月 4 日] 标题：SRRM：用于评估小型语言模型长上下文记忆的检索-重构基准作者：Christopher Alberto Hamonangan。

Abstract: Small Language Models (SLMs) are increasingly deployed on consumer-grade hardware, yet their ability to preserve information across progressively longer contexts remains insufficiently understood. Existing benchmarks primarily evaluate retrieval-oriented tasks, providing limited insight into whether successful retrieval reflects comprehensive long-context memory preservation.

摘要：小型语言模型（SLMs）正越来越多地部署在消费级硬件上，但它们在处理日益增长的上下文时保持信息的能力仍未得到充分理解。现有的基准测试主要评估面向检索的任务，对于成功的检索是否反映了全面的长上下文记忆保持能力，提供的见解有限。

To address this limitation, this paper introduces Simple Retrieval-Reconstruction Memory (SRRM), a lightweight and fully reproducible benchmark that jointly evaluates Targeted Retrieval and Global Information Reconstruction. This study also proposes the Memory Preservation Ratio (MPR), a benchmark-level metric that quantifies the relationship between retrieval and reconstruction performance.

为了解决这一局限性，本文引入了简单检索-重构记忆（SRRM），这是一个轻量级且完全可复现的基准测试，旨在联合评估目标检索和全局信息重构。本研究还提出了记忆保持率（MPR），这是一种基准层面的指标，用于量化检索性能与重构性能之间的关系。

SRRM was evaluated using a synthetic dataset on three Qwen2.5 models (0.5B, 1.5B, and 3B parameters) across context loads ranging from 10 to 100 factual statements. Experimental results show that reconstruction performance deteriorated substantially faster than retrieval accuracy as contextual information increased, indicating that retrieval alone is insufficient to characterize long-context memory preservation.

SRRM 使用合成数据集在三个 Qwen2.5 模型（0.5B、1.5B 和 3B 参数）上进行了评估，上下文负载范围从 10 条到 100 条事实陈述不等。实验结果表明，随着上下文信息的增加，重构性能的下降速度明显快于检索准确率，这表明仅靠检索不足以表征长上下文记忆的保持能力。

Among the evaluated models, Qwen2.5-1.5B achieved the highest retrieval accuracy (86.7%), reconstruction performance (45.7%), and MPR (0.53), outperforming the larger Qwen2.5-3B model (72.2%, 18.5%, and 0.26, respectively). These findings demonstrate that increasing parameter count does not necessarily improve long-context memory preservation.

在所评估的模型中，Qwen2.5-1.5B 实现了最高的检索准确率（86.7%）、重构性能（45.7%）和 MPR（0.53），优于更大的 Qwen2.5-3B 模型（分别为 72.2%、18.5% 和 0.26）。这些发现表明，增加参数数量并不一定会提高长上下文记忆的保持能力。

Overall, SRRM provides a reproducible and extensible benchmark for systematically evaluating long-context memory while revealing complementary memory behaviors that remain hidden under retrieval-only evaluation frameworks.

总体而言，SRRM 提供了一个可复现且可扩展的基准，用于系统地评估长上下文记忆，同时揭示了在仅检索评估框架下被隐藏的互补性记忆行为。