Multi-level context Modeling for consistent expert selection in Mixture-of-Experts

混合专家模型中用于一致性专家选择的多层上下文建模

Abstract: Mixture-of-Experts (MoE) enables efficient scaling of Transformer models by routing tokens to a small subset of experts. However, existing routers typically condition expert selection on shallow or isolated token representations, which often produce unstable and semantically inconsistent routing decisions across layers.

摘要： 混合专家模型（MoE）通过将 Token 路由到一小部分专家，实现了 Transformer 模型的高效扩展。然而，现有的路由器通常基于浅层或孤立的 Token 表示来进行专家选择，这往往会导致跨层路由决策的不稳定和语义不一致。

In this work, we revisit expert selection from a representation perspective and identify context incompleteness as a key bottleneck limiting effective expert specialization. To address this issue, we propose Multi-level Context Fusion MOE (MCF-MOE), a framework that constructs context-aware representations by integrating complementary signals from cross-layer semantic aggregation and local token-level interactions, enabling more informative and consistent expert selection.

在这项工作中，我们从表示学习的角度重新审视了专家选择问题，并指出上下文信息的不完整性是限制专家有效专业化的关键瓶颈。为了解决这一问题，我们提出了多层上下文融合 MoE（MCF-MOE）框架。该框架通过整合来自跨层语义聚合和局部 Token 级交互的互补信号，构建了具有上下文感知能力的表示，从而实现了信息量更大且更具一致性的专家选择。

Experiments on language modeling and understanding benchmarks demonstrate that MCF-MOE consistently improves routing consistency and downstream performance over strong MoE baselines, highlighting the importance of contextual completeness in expert routing. The code is available at this https URL.

在语言建模和理解基准测试上的实验表明，与强大的 MoE 基线相比，MCF-MOE 持续提升了路由一致性和下游任务性能，凸显了上下文完整性在专家路由中的重要性。代码已在指定链接开源。