Token Utility for On-Policy Distillation via Local Trust-Region Optimization
Computer Science > Machine Learning arXiv:2610.18883 (cs) [Submitted on 1 Oct 2026] Title: Token Utility for On-Policy Distillation via Local Trust-Region Optimization Authors: Narayanan Sadagopan, Yilan Chen, Vijay Lingam.
计算机科学 > 机器学习 arXiv:2610.18883 (cs) [提交于 2026 年 10 月 1 日] 标题：通过局部信赖域优化进行策略内蒸馏的 Token 效用 作者：Narayanan Sadagopan, Yilan Chen, Vijay Lingam。
Abstract: Token-selection methods for on-policy distillation (OPD) prioritize student-visited positions using diverse signals, including uncertainty, teacher-student disagreement, and mismatch direction. We instead derive token importance from the distillation objective itself: how much can each position’s loss decrease under an equal local KL budget on the student’s predictive distribution?
摘要：用于策略内蒸馏（OPD）的 Token 选择方法利用多种信号（包括不确定性、师生分歧和失配方向）来优先处理学生访问过的位置。我们转而从蒸馏目标本身推导 Token 的重要性：在学生预测分布的相同局部 KL 预算下，每个位置的损失能减少多少？
The answer is the utility $s_t=g_t^\top F_t^\dagger g_t$, where $g_t$ is the logit gradient and $F_t$ is the categorical Fisher matrix. For reverse-KL it equals the student-weighted variance of the student-teacher log-probability ratio, and for forward-KL it equals the Pearson $\chi^2$ divergence, so the appropriate utility follows from the choice of objective.
答案是效用 $s_t=g_t^\top F_t^\dagger g_t$，其中 $g_t$ 是 Logit 梯度，$F_t$ 是分类 Fisher 矩阵。对于反向 KL，它等于师生对数概率比的学生加权方差；对于前向 KL，它等于 Pearson $\chi^2$ 散度，因此适当的效用取决于目标的选择。
On shared training checkpoints, several independently motivated criteria, including student entropy, KL disagreement, and TIP, concentrate their importance on nearly the same positions as the derived utility at a 50% selection fraction, while diverging at stricter fractions and for direction-specific criteria.
在共享训练检查点上，几种独立提出的准则（包括学生熵、KL 分歧和 TIP）在 50% 的选择比例下，将其重要性集中在与推导出的效用几乎相同的位置上，但在更严格的比例和特定方向的准则下则会出现分歧。
We instantiate the utility as Fisher Utility Weighting (FUW), a continuous, mean-normalized token weighting. Across seven mathematical reasoning benchmarks and three Qwen3 student-teacher pairs, FUW improves over matched uniform OPD under both reverse-KL (1.35-4.0 macro points) and forward-KL (1.08-2.71 macro points), and is competitive with existing selective methods using a single closed-form score derived from the training objective.
我们将该效用实例化为 Fisher 效用加权（FUW），这是一种连续的、均值归一化的 Token 加权方法。在七个数学推理基准和三个 Qwen3 师生对上，FUW 在反向 KL（1.35-4.0 宏观分）和前向 KL（1.08-2.71 宏观分）下均优于匹配的均匀 OPD，并且通过使用从训练目标导出的单一闭式评分，与现有的选择性方法相比具有竞争力。