The Symmetry That Breaks Neural Network Averaging
The Symmetry That Breaks Neural Network Averaging
打破神经网络平均化的对称性
If you have ever tried neural network weight averaging by training the same neural network architecture twice on the same data, changing nothing but the random seed, you have probably assumed the two results were basically interchangeable. Both runs converge. Both reach the same loss. So you average the two weight vectors, expecting something at least as good as either one alone. 如果你曾尝试通过在相同数据上训练两次相同的神经网络架构（仅改变随机种子）来进行权重平均，你可能认为这两个结果基本是可以互换的。两次训练都收敛了，也都达到了相同的损失值。因此，你对这两个权重向量取平均，期望得到一个至少和其中任何一个一样好的结果。
The average is worse. Often much worse. There were no problems during the training process; this is a structural feature of neural networks and can be directly deduced from a property known as permutation symmetry. The same principle also accounts for why model merging succeeds when it does and fails when it fails, a question which, in 2026, is at the heart of practical LLM engineering, whether one is dealing with model soups or federated averaging. 但平均后的结果更差，通常差得多。训练过程中并没有出现任何问题；这是神经网络的一种结构性特征，可以直接从一种称为“置换对称性”（permutation symmetry）的属性中推导出来。同样的原理也解释了模型合并为何有时成功、有时失败——这个问题在 2026 年是实用大模型（LLM）工程的核心，无论是处理模型汤（model soups）还是联邦平均（federated averaging）。
Here is a way to hold the idea before the notation arrives. Think of two trained networks as two spreadsheets describing the same report, except the columns are in a different order. Model A’s “column 3” might hold what Model B calls “column 7.” Both spreadsheets are correct. Both add up to the same totals. But average them cell by cell without first lining the columns up, and you’re averaging revenue with headcount. That’s what naive weight averaging does to a neural network, and it’s worth keeping that image in mind through everything below. 在引入符号之前，这里有一个理解该概念的方法。将两个训练好的网络想象成描述同一份报告的两个电子表格，只是列的顺序不同。模型 A 的“第 3 列”可能对应模型 B 的“第 7 列”。两个表格都是正确的，总计结果也相同。但如果不先对齐列就逐单元格取平均，你实际上是在把“收入”和“员工人数”进行平均。这就是朴素权重平均对神经网络所做的事情，在阅读下文时，请务必记住这个形象的比喻。
Most introductions to neural networks stay in function space: what the network computes, how an activation function bends a line into a curve. This article stays in parameter space: what the set of good solutions actually looks like, and what that geometry costs you. 大多数神经网络入门介绍都停留在函数空间：网络计算什么，激活函数如何将直线弯曲成曲线。本文则停留在参数空间：好的解集实际上是什么样子的，以及这种几何结构会让你付出什么代价。
Why Neural Networks Are Non-Convex: Adaptive Basis Functions and Permutation Symmetry
为什么神经网络是非凸的：自适应基函数与置换对称性
Many classic machine learning models have the form: where $\phi(x) = [\phi_1(x), \dots, \phi_m(x)]^\top$ is a set of basis functions, and $a$ is a coefficient vector. Polynomial regression uses $\phi_\ell(x) = x^{\ell-1}$. Kernel ridge regression takes an implicit, effectively infinite basis defined by the kernel $k(x, x’) = \sum_\ell \lambda_\ell \psi_\ell(x) \psi_\ell(x’)$. 许多经典的机器学习模型具有以下形式：其中 $\phi(x) = [\phi_1(x), \dots, \phi_m(x)]^\top$ 是一组基函数，$a$ 是系数向量。多项式回归使用 $\phi_\ell(x) = x^{\ell-1}$。核岭回归采用由核函数 $k(x, x’) = \sum_\ell \lambda_\ell \psi_\ell(x) \psi_\ell(x’)$ 定义的隐式且实际上无限的基。
In both cases, we are only learning the coefficients $a$, and the loss is a convex quadratic. One minimum, closed-form solution, no random seeds. 在这两种情况下，我们只学习系数 $a$，且损失函数是一个凸二次函数。只有一个极小值，有闭式解，不需要随机种子。
A neural network changes one thing: it makes the basis functions themselves learnable. Each neuron is an adaptive basis function, and the network learns both the basis and the coefficients at the same time by minimizing the same kind of squared-error loss. 神经网络改变了一点：它使基函数本身变得可学习。每个神经元都是一个自适应基函数，网络通过最小化同种类型的平方误差损失，同时学习基和系数。
That is the whole idea. A neuron is an adaptive basis function. We no longer have to guess which features matter. The network fits the features and the coefficients at the same time. This is the core trade-off. Non-convexity is not an accident of ReLU activation. It is the price of adaptivity. Fix the basis, and we get convexity. Learn the basis, and we lose it. There is no third option. 这就是核心思想。神经元就是自适应基函数。我们不再需要猜测哪些特征重要。网络同时拟合特征和系数。这就是核心权衡。非凸性并非 ReLU 激活函数的偶然产物，而是自适应性的代价。固定基，我们得到凸性；学习基，我们失去凸性。没有第三种选择。
It also explains where the “column order” problem comes from. Polynomial regression’s columns are fixed by convention — $x, x^2, x^3$, in that order, agreed on in advance. A neural network’s columns are learned along with everything else, which means there’s no agreed-upon order for them to land in. That missing convention is the entire source of permutation symmetry. 这也解释了“列顺序”问题的由来。多项式回归的列是按照惯例固定的——即 $x, x^2, x^3$ 这种预先商定好的顺序。而神经网络的列是与其他参数一起学习的，这意味着它们没有预先商定的顺序。这种惯例的缺失正是置换对称性的全部来源。
Where the non-convexity comes from: Permutation Symmetry
非凸性的来源：置换对称性
Consider a two-neuron network with one input and ReLU activations: $f(x) = \text{ReLU}(x - w_1) + \text{ReLU}(x - w_2)$. The loss function is symmetric: swapping $w_1$ and $w_2$ gives exactly the same function, and therefore the same loss. So if $(w_1, w_2)$ is a global minimum, then $(w_2, w_1)$ is also a global minimum. 考虑一个具有一个输入和 ReLU 激活函数的双神经元网络：$f(x) = \text{ReLU}(x - w_1) + \text{ReLU}(x - w_2)$。损失函数是对称的：交换 $w_1$ 和 $w_2$ 会得到完全相同的函数，因此损失也相同。所以，如果 $(w_1, w_2)$ 是一个全局极小值，那么 $(w_2, w_1)$ 也是一个全局极小值。
Now, what happens at the midpoint of these two minimizers? By symmetry, the midpoint is $((w_1+w_2)/2, (w_1+w_2)/2)$ i.e., both neurons have the same parameters. A network with two identical neurons has the expressive power of one neuron, not two. Unless one of the original neurons was doing nothing, the midpoint cannot be optimal. 现在，在这两个极小值的中间点会发生什么？根据对称性，中间点是 $((w_1+w_2)/2, (w_1+w_2)/2)$，即两个神经元具有相同的参数。一个拥有两个相同神经元的网络，其表达能力等同于一个神经元，而非两个。除非原始神经元中有一个是无效的，否则中间点不可能是最优的。
That’s the whole story. The loss landscape isn’t just bumpy; it contains many exact copies of the same basin, separated by barriers. For a hidden layer with $m$ units, there are $m!$ equivalent orderings. A 512-unit layer gives roughly $512! \approx 10^{1166}$ copies of every solution per layer. That is $512!$ different ways to shuffle the same set of columns, each one an equally valid spreadsheet of the same report. 这就是全部真相。损失地形不仅是崎岖不平的；它包含了许多由障碍隔开的相同盆地的精确副本。对于一个拥有 $m$ 个单元的隐藏层，存在 $m!$ 种等价的排序。一个 512 单元的层每层大约有 $512! \approx 10^{1166}$ 个解的副本。这意味着有 $512!$ 种不同的方式来重排同一组列，每一种都是同一份报告中同样有效的电子表格。
A Simple Two-Neuron Example of Weight Averaging Failure
权重平均失败的一个简单双神经元示例
The naive average is 6.5× worse than either input. Permute B’s neurons first; reorder its columns to match A’s, and the average snaps back to a perfect model. Nothing about what B computes changed. Only the labels did. 朴素平均的结果比任何一个输入模型都要差 6.5 倍。如果先对 B 的神经元进行置换，重新排序其列以匹配 A，平均后的结果会瞬间恢复为一个完美的模型。B 计算的内容没有任何改变，改变的仅仅是标签（顺序）。