Recurrent Self-Improvement: Dynamic Cross-Loop On-Policy Distillation for Looped Language Models

Computer Science > Machine Learning arXiv:2610.10623 (cs) [Submitted on 7 Oct 2026] Title: Recurrent Self-Improvement: Dynamic Cross-Loop On-Policy Distillation for Looped Language Models Authors: Yi Wang, Rui Qian, Yu Li, Haoyang Yao, Wenjie Wang.

计算机科学 > 机器学习 arXiv:2610.10623 (cs) [提交于 2026 年 10 月 7 日] 标题：循环自我提升：循环语言模型的动态跨循环同策略蒸馏 作者：Yi Wang, Rui Qian, Yu Li, Haoyang Yao, Wenjie Wang。

Abstract: Looped Language Models (LoopLMs) offer a parameter efficient approach to scaling reasoning by reusing shared parameters across recurrent computation steps. Despite their promise, effective post-training of LoopLMs remains challenging. Existing approaches either provide reward based supervision that is sparse or costly to extend across loops, or rely on external teachers or privileged information, leading to limited teacher availability or teacher-student context mismatch.

摘要：循环语言模型 (LoopLMs) 提供了一种参数高效的方法，通过在循环计算步骤中重用共享参数来扩展推理能力。尽管前景广阔，但 LoopLMs 的有效后训练仍然具有挑战性。现有方法要么提供基于奖励的监督，这种监督在跨循环扩展时稀疏或成本高昂，要么依赖外部教师或特权信息，导致教师可用性受限或师生上下文不匹配。

To address these limitations, we introduce LoopOPD, a cross-loop on-policy distillation framework that uses additional recurrent computation within a LoopLM as its own source of supervision. LoopOPD uses a frozen terminal loop policy as a compute privileged teacher for an intermediate loop student on student generated rollouts, providing dense supervision without an external teacher or privileged information.

为了解决这些局限性，我们引入了 LoopOPD，这是一个跨循环同策略蒸馏框架，它利用 LoopLM 内部额外的循环计算作为其自身的监督来源。LoopOPD 使用冻结的终端循环策略作为计算特权教师，针对学生生成的展开序列对中间循环学生进行指导，从而在无需外部教师或特权信息的情况下提供密集监督。

We further propose Dynamic LoopOPD (D-LoopOPD), which continually refreshes the terminal loop teacher as the shared model parameters are updated, enabling recurrent self-improvement. We characterize how distillation updates propagate across loop depths and derive sufficient conditions under which a single update yields simultaneous local improvement at both loop depths.

我们进一步提出了动态 LoopOPD (D-LoopOPD)，它随着共享模型参数的更新不断刷新终端循环教师，从而实现循环自我提升。我们刻画了蒸馏更新如何在循环深度之间传播，并推导出了在单次更新下同时在两个循环深度产生局部改进的充分条件。

Experiments on Ouro-Thinking models show that LoopOPD improves mathematical reasoning, while D-LoopOPD yields further gains through dynamic teacher updates. Despite being trained only on mathematical data, the resulting models also improve on general reasoning and code generation benchmarks, demonstrating that recurrent computation can serve as an effective source of supervision for LoopLMs.

在 Ouro-Thinking 模型上的实验表明，LoopOPD 提高了数学推理能力，而 D-LoopOPD 通过动态教师更新获得了进一步的提升。尽管仅在数学数据上进行了训练，但所得模型在通用推理和代码生成基准测试中也有所改进，这证明了循环计算可以作为 LoopLMs 的有效监督来源。