When and How Does Muon Precondition? The Role of Structured Gradient Noise
Computer Science > Machine Learning arXiv:2610.31728 (cs) [Submitted on 22 Oct 2026] Title: When and How Does Muon Precondition? The Role of Structured Gradient Noise Authors: Xiaoran Chen, Binghui Li, Liwei Wang, Pinyan Lu.
计算机科学 > 机器学习 arXiv:2610.31728 (cs) [提交于 2026 年 10 月 22 日] 标题：Muon 何时以及如何进行预处理？结构化梯度噪声的作用 作者：Xiaoran Chen, Binghui Li, Liwei Wang, Pinyan Lu。
Abstract: Muon has become a popular optimizer for large language model training, yet its theoretical foundations remain poorly understood. In this work, we study how gradient noise geometry shapes Muon’s preconditioning benefits. We first show, theoretically and empirically, that under the classical isotropic noise assumption, Muon offers no acceleration beyond a rescaling of the learning rate.
摘要：Muon 已成为大语言模型训练中一种流行的优化器，但其理论基础仍未得到充分理解。在这项工作中，我们研究了梯度噪声的几何结构如何塑造 Muon 的预处理优势。我们首先从理论和实证两方面证明，在经典的各向同性噪声假设下，Muon 除了对学习率进行重缩放外，无法提供额外的加速。
This motivates us to investigate structured noise arising from mini-batch sampling in associative memory with heterogeneous task frequencies. We establish an exact balancing threshold for two tasks and extend the analysis to multiple tasks, revealing how task coverage and gradient reliability jointly shape learning.
这促使我们研究在具有异构任务频率的联想记忆中，由小批量采样产生的结构化噪声。我们为两个任务建立了一个精确的平衡阈值，并将分析扩展到多个任务，揭示了任务覆盖率和梯度可靠性如何共同塑造学习过程。
Our analysis shows that Muon can soften the dependence on task frequency from linear to square-root, helping balance the learning of frequent and rare tasks. We further identify rotation-induced negative transfer in SignSGD that Muon avoids, and explain how momentum reduces sampling noise while tracking changing gradients.
我们的分析表明，Muon 可以将对任务频率的依赖从线性降低到平方根级别，从而有助于平衡频繁任务和稀有任务的学习。我们进一步识别了 SignSGD 中由旋转引起的负迁移，Muon 避免了这种情况，并解释了动量如何在跟踪变化梯度的同时减少采样噪声。
Controlled numerical experiments validate these mechanisms, and a preliminary LLM pretraining study illustrates the practical promise of fast-slow momentum.
受控数值实验验证了这些机制，一项初步的 LLM 预训练研究展示了快慢动量的实际应用前景。