Scaling Multi-Teacher Distillation for Digital Pathology

Computer Science > Computer Vision and Pattern Recognition arXiv:2610.12574 (cs) [Submitted on 8 Oct 2026] Title: Scaling Multi-Teacher Distillation for Digital Pathology Authors: Sofiène Boutaj, Pierre Marza, Varun Belagali, Dimitris Samaras, Maria Vakalopoulou, Stergios Christodoulidis.

计算机科学 > 计算机视觉与模式识别 arXiv:2610.12574 (cs) [提交于 2026 年 10 月 8 日] 标题：数字病理学中多教师蒸馏的扩展 作者：Sofiène Boutaj, Pierre Marza, Varun Belagali, Dimitris Samaras, Maria Vakalopoulou, Stergios Christodoulidis。

Abstract: Multi-teacher distillation allows transferring knowledge from multiple “teacher” networks to a single “student” network. This method is promising in fields such as digital pathology, where many powerful foundation models were proposed recently. However, as we show in this paper, the standard multi-teacher distillation approach reacts poorly to an increase in the number of teachers used to train a student encoder.

摘要：多教师蒸馏允许将知识从多个“教师”网络转移到单个“学生”网络。这种方法在数字病理学等领域前景广阔，因为该领域最近提出了许多强大的基础模型。然而，正如我们在本文中所展示的那样，标准的多教师蒸馏方法在用于训练学生编码器的教师数量增加时，表现不佳。

We demonstrate that learning teacher-specific representations is a key to scaling in MTD. Importantly, different design choices, such as learnable teacher tokens, a tailored attention scheme, additional mixture-of-experts layers and a contrastive loss, are proposed to better learn such teacher-specific representations.

我们证明，学习特定于教师的表征是实现多教师蒸馏（MTD）扩展的关键。重要的是，我们提出了不同的设计选择，例如可学习的教师标记、定制的注意力机制、额外的专家混合层以及对比损失，以更好地学习这些特定于教师的表征。

We show that our method better scales with respect to the number of considered teachers, allowing us to train compact student encoders, up to 10x more efficient than larger teacher foundation models, while matching their performance and even outperforming them on a large set of tile-level and slide-level tasks.

我们展示了我们的方法在考虑教师数量方面具有更好的扩展性，使我们能够训练紧凑的学生编码器，其效率比大型教师基础模型高出 10 倍，同时在大量切片级和载玻片级任务中匹配甚至超越了它们的性能。

We conduct a thorough empirical validation, evaluating more than 10 foundation models on 39 tasks spanning tile and slide levels. As a result, we release a new collection of strong and efficient foundation models, named OMNI, trained from the knowledge of 10 state-of-the-art foundation models.

我们进行了彻底的实证验证，在涵盖切片和载玻片级别的 39 个任务上评估了 10 多个基础模型。最终，我们发布了一系列名为 OMNI 的强大且高效的新型基础模型，这些模型是根据 10 个最先进的基础模型的知识训练而成的。

We finally conduct an analysis of the learned teacher-specific representations, highlighting their complementarity and explaining why they can be easily aggregated at downstream time.

最后，我们对所学习的特定于教师的表征进行了分析，强调了它们的互补性，并解释了为什么它们可以在下游阶段轻松聚合。