RAPID: Reliability-Aware Pair Importance Distillation

Abstract: Inter-example relational distillation transfers a teacher’s representation geometry by matching relations among examples within a mini-batch. Computing all pairs has quadratic complexity in the batch size, whereas uniform subsampling may use a limited relation budget inefficiently.

摘要： 样本间关系蒸馏通过匹配小批量（mini-batch）内样本之间的关系，来迁移教师模型的表征几何结构。计算所有样本对的复杂度与批大小呈二次方关系，而均匀子采样可能会导致有限的关系预算使用效率低下。

We introduce Reliability-Aware Pair Importance Distillation, or RAPID, which separates a reliability-gated relational target from a full-support adaptive pair proposal. Reliability determines which teacher relations are emphasized, while calibrated teacher entropy and detached student-teacher residuals determine which relations are evaluated.

我们引入了“可靠性感知对重要性蒸馏”（Reliability-Aware Pair Importance Distillation，简称 RAPID），它将可靠性门控的关系目标与全支持自适应对提议分离开来。可靠性决定了哪些教师关系应被强调，而校准后的教师熵和分离的学生-教师残差则决定了哪些关系应被评估。

Exact inverse proposal correction makes the loss and gradient estimators conditionally unbiased with respect to the gated mini-batch target. We evaluate RAPID in two text classification settings: AG News with BERT-to-DistilBERT distillation using three paired seeds and a relation budget of 256, and SST-2 with DistilBERT-to-DistilBERT distillation using three paired seeds and a relation budget of 64.

精确的逆提议校正使得损失和梯度估计量相对于门控小批量目标而言是条件无偏的。我们在两个文本分类设置中评估了 RAPID：一是使用三个配对种子和 256 个关系预算的 AG News（BERT 到 DistilBERT 蒸馏）；二是使用三个配对种子和 64 个关系预算的 SST-2（DistilBERT 到 DistilBERT 蒸馏）。

Reliability-gated relational distillation achieves the highest observed mean student accuracy on both datasets: 94.285 ± 0.054% on AG News and 88.800 ± 0.532% on SST-2. RAPID ranks second, achieving 94.241 ± 0.025% and 88.685 ± 0.462%, respectively, compared with 94.154 ± 0.124% and 87.271 ± 0.162% for the cross-entropy baseline.

可靠性门控关系蒸馏在两个数据集上均达到了观测到的最高平均学生准确率：AG News 上为 94.285 ± 0.054%，SST-2 上为 88.800 ± 0.532%。RAPID 位列第二，分别达到 94.241 ± 0.025% 和 88.685 ± 0.462%，而交叉熵基准模型的结果分别为 94.154 ± 0.124% 和 87.271 ± 0.162%。

Pilot evaluations are counted toward the same total budget as the main relation evaluations. Across both settings, the gated target yields the highest mean accuracy, while the adaptive proposal remains within seed-level variation. These results support the modular view that target reliability and evaluation priority are separable design dimensions.

试点评估与主要关系评估计入相同的总预算。在两种设置中，门控目标均产生了最高的平均准确率，而自适应提议保持在种子水平的变异范围内。这些结果支持了一种模块化观点，即目标可靠性和评估优先级是可分离的设计维度。