One Model to Loop Them All: Elastic Latent Recurrence for Language Models

Title: One Model to Loop Them All: Elastic Latent Recurrence for Language Models

标题：One Model to Loop Them All: Elastic Latent Recurrence for Language Models（一个模型循环所有：语言模型的弹性潜在递归）

Abstract: A latent recurrent language model applies a shared block of layers repeatedly, so one checkpoint can trade inference compute for accuracy by changing the loop count. At one loop, however, such models typically fall well short of a non-recurrent model of the same size, a one-loop tax that makes a separate non-recurrent model the better choice whenever latency matters.

摘要：潜在递归语言模型会重复应用共享的层块，因此单个检查点可以通过改变循环次数来权衡推理计算量与准确性。然而，在单次循环时，此类模型通常远逊于相同规模的非递归模型；这种“单循环税”使得在延迟敏感的情况下，独立的非递归模型成为更好的选择。

We present Elastic Latent Recurrence (ELR), a recipe for training looped language models without this tax. At one loop, an ELR model costs the same to serve as a non-recurrent model of the same size and, trained on the same data, reaches similar accuracy. Each additional loop then buys more accuracy for more inference compute.

我们提出了弹性潜在递归（ELR），这是一种训练循环语言模型且无需支付此“税”的方法。在单次循环时，ELR 模型的服务成本与相同规模的非递归模型相同，且在相同数据训练下能达到相似的准确性。随后，每增加一次循环，都能以更多的推理计算量换取更高的准确性。

Three choices make this work: a gated state update with a learned step size, loop counts mixed within each optimizer update, and conditioning each pass on its loop index and budget. The first two preserve one-loop accuracy, and the third lowers loss at every additional loop.

三个关键选择促成了这一成果：带有学习步长的门控状态更新、在每次优化器更新中混合循环次数，以及根据循环索引和预算对每次传递进行条件化。前两项保留了单循环的准确性，而第三项则降低了每次额外循环的损失。

Mixed depths can be introduced late in pretraining to limit the added training cost, and with zero-initialized conditioning added at the switch, this late schedule reaches lower loss than mixed-depth training from scratch. In MoE models, reusing first-loop expert routes on later loops cuts mixed-depth update time by 11.7% with no measurable change in accuracy.

混合深度可以在预训练后期引入，以限制增加的训练成本；通过在切换点添加零初始化条件，这种后期调度比从头开始的混合深度训练能达到更低的损失。在混合专家（MoE）模型中，在后续循环中重用首轮循环的专家路由，可将混合深度更新时间缩短 11.7%，且准确性无明显变化。

At inference, keeping only the last loop’s attention KV cache holds attention KV memory at its one-loop level while retaining most of the gain from additional loops. On 4B dense and 30B-A3B MoE hybrid models, ELR at one loop is 0.52 points below and 1.04 points above its matched baseline on a ten-task average, and four loops add 7.25 and 4.75 points.

在推理时，仅保留最后一次循环的注意力 KV 缓存，可以将注意力 KV 内存维持在单循环水平，同时保留额外循环带来的大部分增益。在 4B 稠密模型和 30B-A3B MoE 混合模型上，ELR 在单循环时的十项任务平均分分别比匹配基准低 0.52 分和高 1.04 分，而四次循环则分别增加了 7.25 分和 4.75 分。

Carried through post-training, the recipe yields Koi-4B, a single checkpoint that scores an estimated 18 on the Artificial Analysis Intelligence Index at one loop, matching Qwen3.5-4B, and 29 at four loops.

通过后训练，该方法产出了 Koi-4B 模型。这是一个单一检查点，在单次循环时于人工智能分析指数（Artificial Analysis Intelligence Index）上预估得分为 18 分，与 Qwen3.5-4B 持平；在四次循环时得分为 29 分。