Beyond Policy Discrepancy: Capability-Aware Multi-Teacher On-Policy Distillation

查看原文 / View Original 编辑 / Edit

Computer Science > Machine Learning arXiv:2610.26950 (cs) [Submitted on 18 Oct 2026] Title: Beyond Policy Discrepancy: Capability-Aware Multi-Teacher On-Policy Distillation Authors: Xuanfeng Yang, Le Gui, Jiaming Wang, Xuan Zhang.

计算机科学 > 机器学习 arXiv:2610.26950 (cs) [提交于 2026 年 10 月 18 日] 标题：超越策略差异：能力感知型多教师在线策略蒸馏作者：Xuanfeng Yang, Le Gui, Jiaming Wang, Xuan Zhang。

Abstract: Multi-teacher on-policy distillation (MOPD) integrates domain specialists into a shared student, but determining how strongly each teacher should supervise remains challenging. Existing discrepancy-based allocation can assign strong supervision even when the student already matches or exceeds the teacher’s task performance, since behavioral mismatch does not necessarily imply a remaining capability deficit.

摘要：多教师在线策略蒸馏（MOPD）将领域专家整合到一个共享的学生模型中，但确定每位教师应提供多大程度的监督仍然具有挑战性。现有的基于差异的分配方法即使在学生已经达到或超过教师任务表现时，仍可能分配强监督，因为行为不匹配并不一定意味着存在能力缺陷。

We propose Capability-Aware Multi-Teacher On-Policy Distillation (CA-MOPD), which combines specialist merging with capability-guided allocation to account for what the student has already learned. CA-MOPD first merges reinforcement-learning specialists into a strong student initialization, then uses verifier-estimated teacher—student capability gaps to adjust domain-level distillation strength.

我们提出了能力感知型多教师在线策略蒸馏（CA-MOPD），它结合了专家合并与能力引导分配，以考量学生已经学到的知识。CA-MOPD 首先将强化学习专家合并为强大的学生初始化模型，然后利用验证器估计的师生能力差距来调整领域级的蒸馏强度。

Supervision decreases as capability gaps close and increases when estimated student capability regresses, while dense token-level guidance remains provided by the teachers. Across six benchmarks spanning mathematics, code generation, and instruction following, CA-MOPD achieves an average score of 32.22, outperforming naive MOPD, Open-MOPD, and the merged initialization by 0.86, 0.83, and 0.74 points, respectively.

随着能力差距的缩小，监督力度会减弱；而当估计的学生能力倒退时，监督力度会增加，同时教师仍会提供密集的标记级指导。在涵盖数学、代码生成和指令遵循的六个基准测试中，CA-MOPD 的平均得分为 32.22，分别比原始 MOPD、Open-MOPD 和合并初始化模型高出 0.86、0.83 和 0.74 分。

Controlled comparisons further support the benefits of merged initialization and capability-gap guidance under matched training budgets. Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI).

在匹配的训练预算下，对照比较进一步支持了合并初始化和能力差距引导的优势。学科：机器学习 (cs.LG)；人工智能 (cs.AI)。