What Do Current Systematic Generalization Tasks Miss? A Reasoning-Centered Analysis
What Do Current Systematic Generalization Tasks Miss? A Reasoning-Centered Analysis
当前的系统泛化任务缺失了什么？一项以推理为中心的分析
Systematic generalization, the ability to solve novel problems by recombining known atomic elements, is central to human intelligence but difficult to study rigorously under controlled settings. Existing studies therefore rely on simplifications such as approximately linear action composition, productivity-based tests, and action-explicit goals, which make systematic generalization easier to study but omit some essential aspects of this capability.
系统泛化（Systematic generalization）是指通过重组已知原子元素来解决新问题的能力，它是人类智能的核心，但在受控环境下进行严谨研究却十分困难。因此，现有的研究往往依赖于一些简化手段，例如近似线性的动作组合、基于生产力的测试以及动作明确的目标。这些简化虽然使系统泛化更容易研究，但也忽略了该能力的一些本质方面。
To characterize what these simplifications miss, we adopt a reasoning-centered lens and introduce TranSGrid, a testbed that brings deductive, inductive, and abductive reasoning together within a unified task. Experiments with seven Transformers on 4,800 TranSGrid instances show that all models perform much worse on TranSGrid than on a held-out test set: the largest model solves 79.6% of the test set, but only 55.3% of TranSGrid and 15.8% of the hardest subset.
为了刻画这些简化所缺失的内容，我们采用了以推理为中心的视角，并引入了 TranSGrid。这是一个将演绎推理、归纳推理和溯因推理整合在统一任务中的测试平台。在 4,800 个 TranSGrid 实例上对七种 Transformer 模型进行的实验表明，所有模型在 TranSGrid 上的表现都远差于在留出测试集（held-out test set）上的表现：最大的模型在测试集上的解决率为 79.6%，但在 TranSGrid 上仅为 55.3%，而在最困难的子集上仅为 15.8%。
The gap remains within the training length range, showing that productivity alone is not sufficient to evaluate systematic generalization. Additionally, we reintroduce the other two simplifications into TranSGrid: one variant makes actions compose almost linearly (reducing the inductive demand), the other makes goals action-explicit (reducing the abductive one). In both, solve rates return to roughly the test set level, showing that either simplification alone is enough to reduce TranSGrid to an ordinary held-out test set.
这种差距在训练长度范围内依然存在，这表明仅靠生产力（productivity）不足以评估系统泛化能力。此外，我们将另外两种简化手段重新引入 TranSGrid：一种变体使动作几乎呈线性组合（降低了归纳需求），另一种使目标变得动作明确（降低了溯因需求）。在这两种情况下，解决率都回升到了大致与测试集相当的水平，这表明任何一种简化手段都足以将 TranSGrid 降级为一个普通的留出测试集。
Together, our results show that existing tasks reduce either or both of the inductive and abductive demands, and that comprehensively measuring systematic generalization requires a task that involves all three forms of reasoning.
总之，我们的研究结果表明，现有的任务要么降低了归纳需求，要么降低了溯因需求（或两者兼有），而要全面衡量系统泛化能力，则需要一个涵盖所有三种推理形式的任务。