Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress

超越模仿：通过推理进度过滤策略内蒸馏

On-policy distillation (OPD) has emerged as an effective framework for post-training language models by pairing student-generated trajectories with dense token-level supervision from a teacher. 策略内蒸馏（On-policy distillation, OPD）已成为语言模型后训练的一种有效框架，它通过将学生模型生成的轨迹与来自教师模型的密集标记级监督信息进行配对来实现。

However, OPD implicitly assumes that teacher-derived rewards are an appropriate proxy for reasoning progress, and therefore treats all teacher feedback equally during policy optimization. While in practice, this assumption does not always hold. 然而，OPD 隐含地假设教师导出的奖励是衡量推理进度的合适代理指标，因此在策略优化过程中平等对待所有教师反馈。但在实践中，这一假设并不总是成立。

We observe that teacher-derived rewards often conflict with genuine reasoning progress, as reasoning steps with clear reasoning advancement may still receive lower distillation rewards, simply due to deviation from teacher’s outputs. 我们观察到，教师导出的奖励往往与真实的推理进度相冲突，因为某些具有明显推理进展的步骤，仅仅因为偏离了教师的输出，就可能获得较低的蒸馏奖励。

To address this mismatch, we propose Reasoning-Progress-Aware Reward Filtering for On-Policy Distillation (R2-OPD), which constructs two within-trajectory rankings of reasoning spans, one from teacher-derived rewards and the other from independently estimated progress reward. 为了解决这种不匹配问题，我们提出了“推理进度感知奖励过滤策略内蒸馏”（R2-OPD）。该方法构建了推理片段在轨迹内的两种排名：一种基于教师导出的奖励，另一种基于独立估计的进度奖励。

Distillation rewards are selectively suppressed whenever the two rankings disagree, reducing supervision that conflicts with reasoning progress while preserving effective teacher guidance. 每当这两种排名不一致时，蒸馏奖励就会被选择性地抑制，从而在保留有效教师指导的同时，减少与推理进度相冲突的监督信号。

Our approach shows consistent improvement over standard OPD especially regarding reasoning performances. 我们的方法在推理性能方面表现出比标准 OPD 更为持续的改进。