LRMoE: Demand-Side Load Re-Calibration for Efficient MoE Training on Memory-Constrained Nodes

Sparse Mixture-of-Experts (MoE) models offer scalable capacity for Web applications, but uneven routing creates computation and communication bottlenecks. Existing placement and replication methods redistribute this workload, yet additional expert copies compete with token-related buffers for limited GPU memory.

稀疏混合专家（MoE）模型为 Web 应用提供了可扩展的容量，但负载不均的路由机制造成了计算和通信瓶颈。现有的放置和复制方法虽然能重新分配这些工作负载，但额外的专家副本会与 Token 相关缓冲区争夺有限的 GPU 内存。

We present LRMoE, a memory-aware MoE training system that jointly recalibrates backward demand and balances the retained workload. During forward execution, it plans retention quotas and expert placement under resource constraints while preserving all routed computations.

我们提出了 LRMoE，这是一个内存感知型 MoE 训练系统，它能够联合重新校准反向传播需求并平衡保留的工作负载。在正向执行期间，它会在资源约束下规划保留配额和专家放置，同时保留所有已路由的计算。

During backpropagation, a load recalibration metric estimates each token-expert pair’s contribution to the expert update and selects pairs within these quotas. A capacity-aware routing algorithm distributes retained pairs among eligible instances of their original experts.

在反向传播期间，负载重校准指标会评估每个 Token-专家对对专家更新的贡献，并在配额内选择相应的对。一种容量感知路由算法会将保留的对分配给其原始专家的合格实例。

Physical compaction then forms smaller communication payloads and variable-size grouped GEMMs, reducing backward traffic, computation, and execution-buffer demand. Experiments on Qwen3-30B-A3B and GLM4.5-106B-A12B show that LRMoE reduces cumulative MoE-layer training time to fixed-loss targets by 21.2-44.8% relative to SYMI.

物理压缩随后形成更小的通信负载和可变大小的分组 GEMM，从而减少了反向流量、计算量和执行缓冲区需求。在 Qwen3-30B-A3B 和 GLM4.5-106B-A12B 上的实验表明，与 SYMI 相比，LRMoE 将达到固定损失目标的累积 MoE 层训练时间缩短了 21.2-44.8%。