TEXAS: Task-Expert-Aware Supervision for Downstream Mixture-of-Experts LLM Adaptation

TEXAS：面向下游混合专家大模型适配的任务专家感知监督

Abstract: Mixture-of-Experts (MoE) language models route each token through a small subset of experts, making routing patterns useful for identifying task-relevant experts during downstream adaptation. Yet current approaches have two limitations: task experts are typically identified from aggregate routing statistics that reflect usage rather than association with successful task completion, and task-expert activations remain underexplored as signals for supervision allocation.

摘要： 混合专家（MoE）语言模型通过一小部分专家来处理每个 token，这使得路由模式在下游适配过程中对于识别任务相关专家非常有用。然而，当前的方法存在两个局限性：任务专家通常是根据反映使用情况的聚合路由统计数据来识别的，而非根据与任务成功完成的关联性来识别；此外，任务专家激活作为监督分配信号的作用仍未得到充分探索。

We introduce Task-Expert-Aware Supervision (TEXAS), which combines correctness-conditioned task expert discovery with token-level supervision allocation. TEXAS compares expert activations on instances that the base model solves successfully and those it fails to solve, and retains experts more strongly activated on successful instances.

我们引入了任务专家感知监督（TEXAS），它将基于正确性的任务专家发现与 token 级别的监督分配相结合。TEXAS 通过比较基础模型在成功解决和未能解决的实例上的专家激活情况，保留那些在成功实例中激活程度更高的专家。

During fine-tuning, it upweights answer tokens in failed instances when they activate these experts. TEXAS therefore leverages existing routing behavior without restricting adaptation to a fixed expert subset or imposing an explicit target routing distribution.

在微调过程中，当失败实例中的答案 token 激活这些专家时，它会增加这些 token 的权重。因此，TEXAS 利用了现有的路由行为，而无需将适配限制在固定的专家子集内，也不需要强加明确的目标路由分布。

Across three MoE models and six benchmarks, TEXAS achieves the best or tied-best performance in 17 of 18 settings and improves over the strongest baseline by 1.3—1.5 points on average. Ablations and further analyses validate both the discovered experts and the resulting supervision strategy.

在三个 MoE 模型和六个基准测试中，TEXAS 在 18 个设置中的 17 个里达到了最佳或并列最佳性能，平均比最强的基线模型提高了 1.3 到 1.5 个百分点。消融实验和进一步分析验证了所发现的专家以及由此产生的监督策略的有效性。