From Copying to Composition: How Language Models Develop In-Context Learning

查看原文 / View Original 编辑 / Edit

Computer Science > Computation and Language arXiv:2610.19113 (cs) [Submitted on 13 Oct 2026] Title:From Copying to Composition: How Language Models Develop In-Context Learning Authors:Minglai Yang, Xinyu Guo, Renxiong Wang, Sunjie Hou, Daniel Yue Zhang, Yunzhong He View a PDF of the paper titled From Copying to Composition: How Language Models Develop In-Context Learning, by Minglai Yang and 5 other authors View PDF HTML (experimental)

计算机科学 > 计算与语言 arXiv:2610.19113 (cs) [提交于 2026 年 10 月 13 日] 标题：从复制到组合：语言模型如何发展上下文学习作者：Minglai Yang, Xinyu Guo, Renxiong Wang, Sunjie Hou, Daniel Yue Zhang, Yunzhong He 查看标题为《从复制到组合：语言模型如何发展上下文学习》的论文 PDF，作者为 Minglai Yang 及其他 5 位作者查看 PDF HTML（实验性）

Abstract:In-context learning (ICL) allows language models to adapt their behavior from instructions and demonstrations in a prompt, but accuracy alone does not reveal how contextual information controls their predictions. We study the acquisition and trainability of ICL with a controlled diagnostic that separates copying a nearby demonstration from integrating information across the prompt.

摘要：上下文学习（ICL）允许语言模型根据提示中的指令和演示来调整其行为，但仅凭准确率并不能揭示上下文信息如何控制其预测。我们通过一种受控诊断方法研究了 ICL 的获取和可训练性，该方法将复制附近的演示与整合整个提示中的信息区分开来。

Across 17 models, about a third prefer the copied answer, a few are undecided, and the rest prefer the rule-consistent answer, which goes with higher accuracy and with control by the requested value rather than the closest conflicting example. Checkpoint analyses show that models begin prioritizing task-relevant prompt tokens over conflicting examples long before their accuracy improves beyond chance level.

在 17 个模型中，约有三分之一倾向于复制答案，少数处于不确定状态，其余则倾向于符合规则的答案；后者伴随着更高的准确率，并受所请求数值的控制，而非受最近的冲突示例影响。检查点分析表明，模型在准确率提高到随机水平之前，就已经开始优先处理与任务相关的提示标记，而不是冲突示例。

In matched training experiments, synthetic documents that define endings and then query them shift models toward compositional responses on the controlled diagnostic, whereas fine-tuning on definition exercises built from rewritten real documents improves accuracy when prompts reassign label meanings. The results suggest that ICL is not acquired as a single, uniform capability. Instead, prompt control, answer preference, construction without candidates, and transfer can develop along distinct trajectories.

在匹配的训练实验中，定义结尾并随后进行查询的合成文档使模型在受控诊断中转向组合式响应，而基于重写后的真实文档构建的定义练习进行微调，则在提示重新分配标签含义时提高了准确率。结果表明，ICL 并非作为一种单一、统一的能力被获取。相反，提示控制、答案偏好、无候选构建以及迁移能力可以沿着不同的轨迹发展。