MIITA: Memory-Induced Inference-Time Adaptation for Continual Learning with Small Language Models

MIITA：面向小语言模型持续学习的记忆诱导推理时适应框架

Abstract: Continual learning (CL) is essential for small language models (SLMs) to adapt to evolving real-world needs in resource-constrained deployments. However, directly updating their limited parameter space causes catastrophic forgetting. While memory-based methods naturally address this by decoupling knowledge retention from parameters, existing approaches designed for large language models (LLMs) rely on abundant storage and strong in-context reasoning that SLMs lack.

摘要： 持续学习（CL）对于小语言模型（SLMs）在资源受限的部署环境中适应不断变化的现实需求至关重要。然而，直接更新其有限的参数空间会导致灾难性遗忘。虽然基于记忆的方法通过将知识保留与参数解耦，自然地解决了这一问题，但现有为大语言模型（LLMs）设计的方法依赖于充足的存储空间和强大的上下文推理能力，而这些正是小语言模型所缺乏的。

To address these challenges, we propose MIITA, a Memory-Induced Inference-Time Adaptation framework for supervised CL under constrained storage. MIITA stores supervised experiences as compact correction-direction prototypes with semantic anchors, and retrieves them at inference time using semantic and uncertainty-based cues.

为了应对这些挑战，我们提出了 MIITA，这是一个在受限存储条件下，面向监督式持续学习的记忆诱导推理时适应框架。MIITA 将监督经验存储为带有语义锚点的紧凑修正方向原型，并在推理时利用语义和基于不确定性的线索进行检索。

The retrieved directions are applied through gated temporary hidden-state adaptation, enabling non-destructive reuse of past supervision without backbone updates, prompt extensions, or test-time backpropagation. A local theoretical analysis links this design to first-order loss reduction, uncertainty-guided retrieval, and directional coverage for retaining old-stage knowledge.

检索到的方向通过门控临时隐藏状态适应进行应用，从而在无需更新主干网络、扩展提示词或进行测试时反向传播的情况下，实现对过去监督信息的非破坏性重用。局部理论分析将该设计与一阶损失减少、不确定性引导检索以及用于保留旧阶段知识的方向覆盖联系起来。

Extensive experiments across diverse supervised CL settings show that MIITA consistently improves final performance and mitigates forgetting under fixed memory budgets.

在多种监督式持续学习设置下进行的广泛实验表明，MIITA 在固定的内存预算下，能够持续提升最终性能并减轻遗忘现象。