Depth-Aware Sensitivity Analysis of Mixture-of-Experts Models via Magnitude-Based Expert Masking

基于幅值专家掩码的混合专家模型深度敏感性分析

Abstract: Mixture-of-Experts (MoE) architectures scale large language models (LLMs) while preserving computational efficiency through sparse activation. Despite their widespread adoption, the relative importance of individual MoE layers remains insufficiently characterized, particularly for model compression.

摘要： 混合专家（MoE）架构通过稀疏激活在扩展大型语言模型（LLM）的同时保持了计算效率。尽管 MoE 已被广泛采用，但各个 MoE 层之间的相对重要性仍缺乏充分的表征，特别是在模型压缩领域。

This paper presents a systematic layer-wise sensitivity analysis of the Qwen3.6-35B-A3B model (40 MoE layers, 256 experts per layer, top-8 routing) using magnitude-based expert masking on the XLCoST cross-lingual code translation benchmark. We conduct a multi-phase study spanning 100, 300, and 500 prompt evaluation scales across three H100 GPU servers.

本文针对 Qwen3.6-35B-A3B 模型（包含 40 个 MoE 层，每层 256 个专家，采用 top-8 路由）进行了系统的逐层敏感性分析，并在 XLCoST 跨语言代码翻译基准测试上使用了基于幅值的专家掩码技术。我们在三台 H100 GPU 服务器上进行了跨越 100、300 和 500 个提示词评估规模的多阶段研究。

Our central finding is that layer sensitivity is strongly depth-dependent: early layers (0-9) and middle layers (10-29) are highly fragile to expert masking, while late layers (30-39), and especially very-late layers (35-39), tolerate aggressive masking of low-magnitude experts.

我们的核心发现是，层敏感性具有强烈的深度依赖性：早期层（0-9 层）和中间层（10-29 层）对专家掩码非常敏感，而后期层（30-39 层），尤其是极后期层（35-39 层），则能够容忍对低幅值专家进行激进的掩码处理。

Flat all-layer masking at 30% retains only 150/300 Good+Similar outputs at 300-prompt scale, whereas late-focused policies retain 249-255/300 while masking 640-1,145 experts. On a later 500-prompt held-out validation slice, the narrow very-late policy (layers 35-39 @ 50%) achieves the strongest quality/masked-expert tradeoff among tested candidates, retaining 419/500 Good+Similar outputs while masking only 640 of 10,240 total experts.

在 300 个提示词的规模下，对所有层进行 30% 的统一掩码仅能保留 150/300 的“良好+相似”输出，而侧重于后期层的策略在掩码 640 到 1,145 个专家的情况下，仍能保留 249-255/300 的输出。在后续 500 个提示词的留出验证集上，窄范围的极后期策略（35-39 层 @ 50%）在所有测试方案中实现了最佳的质量与掩码专家数量权衡，在仅掩码 10,240 个专家中的 640 个的情况下，保留了 419/500 的“良好+相似”输出。

We additionally characterize top-k routing width reduction from 8 to 6 active experts per token, which shows a large observed wall-clock reduction on a 100-prompt probe with no Good+Similar loss, though it does not yet compose cleanly with aggressive expert masking. These findings provide an empirical foundation for depth-aware MoE expert masking and establish a practical path toward physical weight surgery, activation-based expert scoring, and training-based recovery.

此外，我们还评估了将每个 token 的 top-k 路由宽度从 8 减少到 6 的效果。在 100 个提示词的探测中，该方法在没有损失“良好+相似”输出的前提下，显著降低了实际运行时间，尽管目前它还无法与激进的专家掩码策略完美兼容。这些发现为深度感知的 MoE 专家掩码提供了实证基础，并为物理权重手术、基于激活的专家评分以及基于训练的恢复开辟了实用路径。