The Drift Contract: Spectral Updates for Depth-Robust Local Learning

漂移契约：用于深度鲁棒局部学习的谱更新

Abstract: Local learning trains each layer with its own auxiliary loss and no global backward pass, which makes layer updates structurally parallel. Two problems have kept it marginal: accuracy degrades as depth grows, and hyperparameters are fragile.

摘要： 局部学习（Local learning）通过为每一层设置独立的辅助损失函数，且无需全局反向传播来训练模型，这使得各层的更新在结构上是并行的。然而，有两个问题限制了其应用：一是随着深度的增加，精度会下降；二是超参数非常脆弱。

We apply Muon-style spectral update geometry (momentum orthogonalization with spectral step scaling) to per-layer local updates, an intersection not previously studied. On CIFAR-10 MLP benchmarks with local linear heads, a single step-size setting is the best value in our tested grids from width 128 to 2048 and from depth 12 to 48, while local Adam requires re-tuning along both axes and still collapses at depth 48 (31.3 percent re-tuned per depth, 19 percent with its depth-12 setting transferred, vs 42.7 percent for the spectral update at its unchanged setting).

我们将 Muon 风格的谱更新几何（动量正交化与谱步长缩放）应用于逐层局部更新，这是一个此前未被研究的交叉领域。在带有局部线性头的 CIFAR-10 MLP 基准测试中，在宽度 128 到 2048、深度 12 到 48 的测试网格中，单一的步长设置即为最优值；相比之下，局部 Adam 优化器需要针对这两个维度进行重新调整，且在深度 48 时仍会崩溃（每层深度重新调整后的准确率为 31.3%，直接迁移深度 12 的设置时为 19%，而谱更新在保持设置不变的情况下达到了 42.7%）。

At five seeds and width 512 the spectral update leads local Adam by a clear margin (48.9 +/- 0.5 vs 46.6 +/- 0.3). Prospectively specified controls attribute the transfer and most of the depth robustness to the spectral geometry itself rather than to any step-size rule on top of it.

在五个随机种子和宽度 512 的条件下，谱更新以明显的优势领先于局部 Adam（48.9 +/- 0.5 对比 46.6 +/- 0.3）。前瞻性的对照实验表明，这种迁移能力和大部分深度鲁棒性归功于谱几何本身，而非其上附加的任何步长规则。

We additionally formulate the step size as a drift contract, lr = epsilon / RMS(input), which bounds each layer’s weight-induced pre-activation change per step, conditioned on its current input. The contract yields a small gain over the best fixed learning rate where that baseline is measured, makes the step size interpretable, and provides a per-layer, input-conditioned drift bound that standard optimizers do not offer.

此外，我们将步长公式化为“漂移契约”（drift contract），即 lr = epsilon / RMS(input)。该契约基于当前输入，限制了每一层权重引起的每步预激活变化。与测量出的最佳固定学习率相比，该契约带来了一定的性能提升，使步长具有可解释性，并提供了标准优化器所不具备的、针对每一层且受输入条件约束的漂移边界。

We report one negative result: with RMSNorm and weight decay in the trunk, the stability benefit of spectral updates accrues to global rather than local training, so the local advantage concentrates precisely where normalization is absent.

我们报告了一个负面结果：当在主干网络中使用 RMSNorm 和权重衰减时，谱更新带来的稳定性优势会向全局训练倾斜而非局部训练，因此局部学习的优势恰恰集中在缺乏归一化的区域。