Is MoE Routing a Huffman Code? Discovering the Frequency-Diversity Law in Chain-of-Thought

MoE 路由是霍夫曼编码吗？揭示思维链中的“频率-多样性定律”

Abstract: Mixture-of-Experts architectures have revolutionized scaling, yet the underlying logic of their routing remains a black box. In this paper, we uncover a fundamental governing principle: MoE routing is not merely selection, but a manifestation of Huffman Coding.

摘要： 混合专家（MoE）架构彻底改变了模型扩展的方式，但其路由背后的逻辑仍然是一个黑箱。在本文中，我们揭示了一个基本的支配原则：MoE 路由不仅仅是选择，更是霍夫曼编码（Huffman Coding）的一种体现。

We introduce the Frequency-Diversity Law, revealing that state-of-the-art models, such as Phi-3.5-MoE and Gemma-4-27B-A4B, spontaneously act as information-theoretic engines. These models allocate sparse expert resources for common tokens while invoking high-diversity expert committees for rare, complex tasks found in chain-of-thought trajectories.

我们引入了“频率-多样性定律”（Frequency-Diversity Law），揭示了诸如 Phi-3.5-MoE 和 Gemma-4-27B-A4B 等最先进的模型，会自发地充当信息论引擎。这些模型为常见 Token 分配稀疏的专家资源，同时针对思维链轨迹中出现的罕见、复杂任务，调用高多样性的专家委员会。

However, we identify a critical redundancy trap in Qwen3.5-35B-A3B: when effective sparsity (k/E_eff) is sufficiently low, load-balancing inadvertently imposes functional redundancy, masking the underlying Huffman efficiency signal.

然而，我们在 Qwen3.5-35B-A3B 中发现了一个关键的冗余陷阱：当有效稀疏度（k/E_eff）足够低时，负载均衡会无意中引入功能性冗余，从而掩盖了潜在的霍夫曼效率信号。

To bridge this gap, we propose Subset Difference Pruning, a surgical strategy to eliminate functional duplicates. We demonstrate that pruning does not degrade reasoning; instead, it unleashes the model’s latent Huffman efficiency, forcing the logic to collapse into streamlined, high-density paths.

为了弥补这一差距，我们提出了“子集差异剪枝”（Subset Difference Pruning），这是一种消除功能重复的精细化策略。我们证明了剪枝并不会降低推理能力；相反，它释放了模型潜在的霍夫曼效率，迫使逻辑收敛为精简、高密度的路径。

Our findings suggest that the next generation of MoEs should move beyond forced load-balancing toward Minimum Description Length (MDL) optimality, assigning shorter expert-routing codes to high-frequency information and longer, more diverse codes to low-frequency information, thereby transforming routing from a heuristic into a principled compression engine.

我们的研究结果表明，下一代 MoE 模型应超越强制的负载均衡，转向最小描述长度（MDL）最优性，即为高频信息分配较短的专家路由编码，为低频信息分配较长、更多样化的编码，从而将路由从一种启发式方法转变为一种有原则的压缩引擎。