Woodpecker Distillation: Weak Models Diagnose Reasoning Bugs in Strong Models
Woodpecker Distillation: Weak Models Diagnose Reasoning Bugs in Strong Models
Woodpecker Distillation：弱模型诊断强模型的推理缺陷
Abstract: Large language models often fail on reasoning tasks despite possessing the capability to solve them. We argue that many such failures arise from localized reasoning bugs in intermediate steps rather than from global incompetence.
摘要： 大型语言模型尽管具备解决推理任务的能力，却往往在这些任务上表现不佳。我们认为，许多此类失败并非源于模型整体能力的缺失，而是源于中间步骤中局部推理缺陷所致。
We show that these bugs are frequently repairable: inserting a short patch generated by a weak probe model after the same strong-model reasoning prefix can redirect the trajectory toward a correct solution. However, this corrective effect is not reliably internalized by directly fine-tuning on weak patches or repaired trajectories, suggesting that the useful signal lies not in the intervention text itself, but in how it reshapes the model’s future reasoning distribution.
我们证明，这些缺陷通常是可以修复的：在强模型的推理前缀后插入由弱探测模型生成的简短补丁，可以引导推理轨迹走向正确的解决方案。然而，通过直接在弱补丁或修复后的轨迹上进行微调，并不能可靠地内化这种纠正效果。这表明，有用的信号并不在于干预文本本身，而在于它如何重塑模型未来的推理分布。
We therefore propose Woodpecker Distillation, a weak-to-strong training framework that learns from contrastive local interventions. Our method contrasts successful and unsuccessful weak-model patches at the same prefix, constructs a corrective teacher distribution from their induced future token predictions, and distills this signal into the strong model.
因此，我们提出了 Woodpecker Distillation，这是一种从对比性局部干预中学习的“弱到强”（weak-to-strong）训练框架。我们的方法对比了同一前缀下弱模型成功与失败的补丁，利用它们诱导的未来 Token 预测构建了一个纠正性的教师分布，并将该信号蒸馏到强模型中。
Experiments on mathematical reasoning benchmarks show that Woodpecker Distillation consistently improves strong-model performance and outperforms direct imitation baselines.
在数学推理基准测试上的实验表明，Woodpecker Distillation 能够持续提升强模型的性能，并优于直接模仿的基准方法。
Paper Details:
- Authors: Dayu Wang, Jiaye Yang, Weikang Li, Jiahui Liang, Yang Li, Deguo Xia, Jizhou Huang
- arXiv ID: 2608.05168
- Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
论文详情：
- 作者： Dayu Wang, Jiaye Yang, Weikang Li, Jiahui Liang, Yang Li, Deguo Xia, Jizhou Huang
- arXiv ID: 2608.05168
- 学科分类： 人工智能 (cs.AI)；计算与语言 (cs.CL)