Sphere Retraction Normalizations
Sphere Retraction Normalizations
Abstract: Residual connections are the de facto mechanism for training deep neural networks stably. Geodesic Normalization (GeoNorm) recasts them on a Riemannian manifold, orthogonalizing each layer output against the current hidden state and applying the resulting update through the Riemannian exponential map. Every hidden state thus keeps a constant $\ell_{2}$-norm, confining the residual stream to a hypersphere.
摘要： 残差连接是稳定训练深度神经网络的事实标准机制。测地线归一化（GeoNorm）将其重构在黎曼流形上，使每一层的输出与当前的隐藏状态正交化，并通过黎曼指数映射应用所得的更新。因此，每个隐藏状态都保持恒定的 $\ell_{2}$ 范数，将残差流限制在一个超球面上。
The exponential map, however, is only one member of a broad family of retraction maps. We show that on the hypersphere this entire family collapses to a single scalar design choice. What distinguishes one retraction from another is only how the magnitude of an update is converted into a rotation angle within the plane spanned by the hidden state and the update. This view places Euclidean residual connections and GeoNorm in one framework.
然而，指数映射只是广泛的收缩映射族中的一员。我们证明，在超球面上，整个映射族可以简化为一个单一的标量设计选择。区分不同收缩映射的唯一因素，在于更新的幅度如何转化为由隐藏状态和更新向量所构成的平面内的旋转角度。这一视角将欧几里得残差连接和 GeoNorm 统一到了同一个框架中。
Instantiating it with the metric projection retraction and the Cayley retraction yields Proj-SpheretNorm and Cay-SpheretNorm, which are exactly norm-preserving yet require only algebraic operations. Both prove to be members of a one-parameter family of angular retractions, $p$-SpheretNorm, whose rotation angle saturates rather than growing without bound. The two methods above are recovered exactly at $p = 1$ and $p = 2$, while the identity map and GeoNorm arise only as limits at either end.
通过度量投影收缩和凯莱（Cayley）收缩进行实例化，我们得到了 Proj-SpheretNorm 和 Cay-SpheretNorm，它们在保持范数的同时仅需代数运算。两者都被证明是单参数角收缩族 $p$-SpheretNorm 的成员，其旋转角度会趋于饱和，而不是无限制地增长。上述两种方法分别在 $p = 1$ 和 $p = 2$ 时精确复现，而恒等映射和 GeoNorm 则仅作为两端的极限情况出现。
On nanoGPT, all three methods outperform existing lightweight deep connection schemes, and the best validation loss is attained at finite $p$, indicating that the exponential map is not the preferred retraction for spherical residual streams but merely one end of a spectrum.
在 nanoGPT 上，这三种方法均优于现有的轻量级深度连接方案，且最佳验证损失出现在有限的 $p$ 值处，这表明指数映射并非球形残差流的首选收缩方式，而仅仅是该谱系的一端。