Where Prefill Expert Reduction Hurts MoE Models: A Few Prompt Tokens Carry Most of the Short-Answer F1 Loss

Computer Science > Computation and Language arXiv:2610.22454 (cs) [Submitted on 15 Oct 2026] Title: Where Prefill Expert Reduction Hurts MoE Models: A Few Prompt Tokens Carry Most of the Short-Answer F1 Loss Authors: Zhiyuan Wang.

计算机科学 > 计算与语言 arXiv:2610.22454 (cs) [提交于 2026 年 10 月 15 日] 标题：预填充专家缩减在何处损害 MoE 模型：少数提示词标记承担了大部分短答案 F1 损失 作者：Zhiyuan Wang。

Abstract: Running prompt tokens through fewer routed experts during prefill is a simple way to make mixture-of-experts (MoE) serving cheaper, yet it is unclear what this damages. We locate the damage with token-, layer- and attention-path interventions on short-answer question answering in four MoE families.

摘要：在预填充过程中通过较少的路由专家运行提示词标记，是降低混合专家（MoE）模型服务成本的一种简单方法，但尚不清楚这会造成什么损害。我们通过对四个 MoE 系列的短答案问答进行标记、层和注意力路径干预，定位了这种损害。

At half the experts the F1 loss of three families is mostly answer form: the model still finds the answer but stops answering briefly. In Qwen3-30B-A3B this loss is made at the two chat-template tokens before the answer, and in DeepSeek-V2-Lite at the first 8-32 prompt tokens, where its instruction sits; by answer likelihood these positions account for about half and a fifth of the damage.

在专家数量减半的情况下，三个系列的 F1 损失主要表现为答案格式问题：模型仍然能找到答案，但会短暂停止回答。在 Qwen3-30B-A3B 中，这种损失发生在答案之前的两个聊天模板标记处；而在 DeepSeek-V2-Lite 中，则发生在指令所在的最初 8-32 个提示词标记处；按答案可能性计算，这些位置分别占了损害的约二分之一和五分之一。

Reduction changes the first token’s attention mass, norm and largest activation by less than 4%, and restoring super experts or massive activations recovers at most about a quarter of the loss, well below keeping the sensitive positions full.

缩减对第一个标记的注意力权重、范数和最大激活值的影响不到 4%，而恢复超级专家或大规模激活最多只能弥补约四分之一的损失，远低于保持敏感位置完整的效果。

On real prompts with about 4% of tokens at full experts, placing them on the task interface keeps F1 within one point of the unmodified model, while placing them at random in the documents is about 3 points worse (two of five prespecified primary comparisons; the others were undecided or not run).

在约 4% 的标记使用完整专家的真实提示中，将它们放置在任务界面上可使 F1 值保持在未修改模型的一个点以内，而将它们随机放置在文档中则会差约 3 个点（这是五个预设主要比较中的两个；其余的未决定或未运行）。

Because the answer boundary always ends a single-turn prompt, and DeepSeek’s loss was no longer detectable once its instruction moved away from the start, the sensitive tokens lay at the prompt ends in every sensitive layout we found, so rules that keep the ends are not distinguished from rules that read application marks.

由于答案边界总是结束单轮提示，且 DeepSeek 的损失在指令移出起始位置后便无法检测到，因此在我们发现的每个敏感布局中，敏感标记都位于提示的末端，因此保留末端的规则与读取应用程序标记的规则无法区分。

Time to first token falls about 14% on 11K-token prompts and not at 2K tokens.

在 11K 标记的提示下，首个标记的生成时间下降了约 14%，而在 2K 标记时则没有下降。