A Shared Learning Rate Is Not a Neutral Control in Selective On-Policy Distillation
A Shared Learning Rate Is Not a Neutral Control in Selective On-Policy Distillation
共享学习率在选择性在线策略蒸馏中并非中性控制变量
Abstract: Selective on-policy distillation trains a student only at the token positions a selector scores highest, and the literature compares selectors under a single shared learning rate—a control chosen to be neutral. We show it is not.
摘要： 选择性在线策略蒸馏（Selective on-policy distillation）仅在选择器评分最高的 Token 位置训练学生模型，而现有文献通常在单一共享学习率下比较不同的选择器——这种控制变量被认为是中性的。我们证明事实并非如此。
Under LoRA on GSM8K (Qwen2.5-1.5B student, 7B teacher), across an 8x learning-rate grid, dense supervision is statistically flat (swing 1.8 pp, p=0.26) while every selective arm moves with the rate: 5.4 pp for a random 5% subset, 6.7 pp for a total-variation selector, up to 17.7 pp for a teachability selector.
在 GSM8K 数据集上使用 LoRA（Qwen2.5-1.5B 学生模型，7B 教师模型）进行实验，在 8 倍学习率网格下，密集监督（dense supervision）在统计上表现平稳（波动 1.8 个百分点，p=0.26），而每一个选择性分支都随学习率变化：随机 5% 子集波动 5.4 个百分点，总变差（total-variation）选择器波动 6.7 个百分点，可教性（teachability）选择器波动高达 17.7 个百分点。
Consequently the dense-versus-selective verdict reads 10.1 pp at lr=1e-4 but 5.1 pp at 5e-5—a 2.0x difference decided by a parameter the protocol treats as scenery—and two of six pairwise significance calls between selectors flip between adjacent rates without any rank inversion.
因此，密集监督与选择性监督的对比结果在学习率为 1e-4 时为 10.1 个百分点，但在 5e-5 时仅为 5.1 个百分点——这种 2.0 倍的差异竟是由一个被实验方案视为“背景板”的参数决定的。此外，在六对选择器两两显著性检验中，有两对在相邻学习率下结论发生反转，且并未出现排序倒置。
We call this selector-rate entanglement and trace it to selection itself rather than step size: AdamW update magnitudes track the rate to within 2.2% despite 15.5x gradient-norm differences across arms.
我们将这种现象称为“选择器-学习率纠缠”（selector-rate entanglement），并将其归因于选择过程本身而非步长：尽管各分支间的梯度范数差异高达 15.5 倍，但 AdamW 更新幅度与学习率的跟踪误差在 2.2% 以内。
A preregistered frozen-scoring ablation (selection scored by the initial student; criterion, budget, and on-policy rollouts unchanged; 12 seeds per cell) shows live scoring adds 3.79+/-1.69 pp of rate sensitivity (p=0.035) while the frozen arm remains significantly entangled (p=0.015): the feedback loop aggravates the phenomenon rather than causing it.
一项预注册的冻结评分消融实验（由初始学生模型进行选择评分；标准、预算和在线策略展开保持不变；每个单元 12 个随机种子）显示，实时评分增加了 3.79+/-1.69 个百分点的学习率敏感度（p=0.035），而冻结分支仍然存在显著的纠缠现象（p=0.015）：反馈循环加剧了这一现象，但并非其根本原因。
Under full fine-tuning at the rates this literature actually uses (1e-6 to 1e-5) the pattern grows: dense itself swings 19.8 pp, the selective arm 49.5 pp, and the verdict ranges from a non-significant +3.6 pp at the published operating point to +34 pp (p=0.005) one notch hotter.
在现有文献实际使用的学习率（1e-6 到 1e-5）下进行全参数微调时，这种模式进一步加剧：密集监督本身波动 19.8 个百分点，选择性分支波动 49.5 个百分点；对比结论从已发表工作中的不显著（+3.6 个百分点）变为更高学习率下的显著（+34 个百分点，p=0.005）。
On MATH-500 the rate dependence does not reproduce under LoRA, scoping that result, while the ~10 pp cost of selective training does. We prescribe reporting the arm x rate matrix, not a shared-rate column, as a precondition for selector comparisons.
在 MATH-500 数据集上，LoRA 训练下并未复现这种学习率依赖性，这界定了该结果的适用范围，但选择性训练带来的约 10 个百分点的性能代价依然存在。我们建议将“分支 x 学习率”矩阵作为选择器比较的前提条件进行报告，而非仅提供单一共享学习率下的结果。