Recursive Language Models Generalize Out of Domain

递归语言模型在域外泛化能力的研究

Abstract: We study when limiting what a language model can see improves learning. We compare standard CoT, the more general learner that reads the full trace, with recursive language models, which restricts itself by solving each subtask in an isolated context.

摘要： 我们研究了限制语言模型的可见信息在何时能提升学习效果。我们将标准的思维链（CoT，即读取完整轨迹的通用学习器）与递归语言模型进行了对比，后者通过在隔离的上下文中解决每个子任务来限制自身。

In-distribution, this generality comes for free: CoT can efficiently simulate the recursive rule, so the IID generalization guarantee changes only by a constant factor, and recursion does not offer much.

在分布内（In-distribution）情况下，这种通用性是“免费”的：CoT 可以高效地模拟递归规则，因此独立同分布（IID）的泛化保证仅会有常数倍的变化，此时递归方法并无太大优势。

But out of domain, CoT can fit training by relying on context outside the current subtask, i.e. a shortcut that breaks once those tokens change; recursive context isolation rules out this failure mode.

但在域外（Out of domain）情况下，CoT 可能会通过依赖当前子任务之外的上下文来拟合训练数据，即一种一旦相关标记（tokens）发生变化就会失效的“捷径”；而递归的上下文隔离机制则排除了这种失效模式。

Even though CoT’s class still covers the recursive rule, simplicity bias picks the shortcut over the truth. Thus, to go beyond distributional accuracy and truly reason, covering the right rule is not enough; this contrasts with classical learning theory.

尽管 CoT 的模型类仍然涵盖了递归规则，但“简单性偏差”（simplicity bias）会使其倾向于选择捷径而非真理。因此，要超越分布准确性并实现真正的推理，仅仅覆盖正确的规则是不够的；这一点与经典学习理论形成了对比。