MILES: Modular Instruction Memory with Learnable Selection for Self-Improving LLM Reasoning

MILES：用于大语言模型自我提升推理的模块化指令记忆与可学习选择机制

Large language models (LLMs) increasingly improve their reasoning at test time via additional computation, yet most existing works treat each problem in isolation. When problems arrive sequentially, accumulating reusable experience across them can further improve performance. 大语言模型（LLMs）正越来越多地通过额外的计算在测试时提升其推理能力，然而大多数现有研究将每个问题视为孤立的个体。当问题按顺序出现时，跨问题积累可复用的经验可以进一步提升性能。

Existing memory-based methods either store whole-solution templates that generalize poorly to novel problems or use heuristic step-level selection that is not optimized for final-answer correctness. Learning selection policies requires large-scale training data and fixed action spaces, making such approaches unsuitable for test-time settings where memory expands incrementally and only limited supervision is available. 现有的基于记忆的方法要么存储难以泛化到新问题的完整解决方案模板，要么使用未针对最终答案正确性进行优化的启发式步骤级选择。学习选择策略需要大规模的训练数据和固定的动作空间，这使得此类方法不适用于记忆增量扩展且监督信息有限的测试时环境。

We propose MILES (Modular Instruction Memory with LEarnable Selection for self-improving LLM reasoning), a framework that dynamically expands step-wise memory and applies correctness-optimized memory composition under realistic test-time constraints. 我们提出了 MILES（用于大语言模型自我提升推理的模块化指令记忆与可学习选择机制），这是一个能够在现实的测试时约束下，动态扩展步骤级记忆并应用针对正确性优化的记忆组合框架。

MILES maintains modular memory units consisting of asymmetric pairs of sub-goal embeddings and sub-instructions, each associated with a learnable selection head. This memory structure enables a coarse-to-fine retrieval mechanism: The coarse level enables memory expansion and collects supervision for training selection heads from confident samples, while the fine stage applies learned selection heads to rerank coarse-level candidates and guide reasoning for uncertain samples. MILES 维护着由子目标嵌入和子指令组成的非对称对模块化记忆单元，每个单元都关联一个可学习的选择头。这种记忆结构实现了一种从粗到细的检索机制：粗粒度层支持记忆扩展，并从置信度高的样本中收集用于训练选择头的监督信息；而细粒度阶段则应用已学习的选择头对粗粒度候选结果进行重排序，并为不确定的样本提供推理指导。

MILES consistently matches or outperforms prior methods while achieving superior accuracy-efficiency tradeoffs. Extensive experiments demonstrate its effectiveness, robustness, and transferability. MILES 在实现卓越的准确性与效率平衡的同时，始终能够达到或超越现有方法。广泛的实验证明了其有效性、鲁棒性和可迁移性。