Data Predictability Shapes Weibull Weight-Scale Growth in Transformer Training

数据可预测性决定了 Transformer 训练中 Weibull 权重尺度的增长

Abstract: A trained transformer’s weight magnitudes can be summarized by a two-parameter Weibull distribution whose shape $k \approx 1.2$ is stable across layers and models, so the scale $\lambda$ carries most training-induced movement. What corpus property sets how much $\lambda$ grows?

摘要： 经过训练的 Transformer 权重幅值可以用双参数 Weibull 分布来概括，其形状参数 $k \approx 1.2$ 在不同层和模型间保持稳定，因此尺度参数 $\lambda$ 承载了训练过程中产生的大部分变化。那么，语料库的哪种属性决定了 $\lambda$ 的增长幅度呢？

Using the bigram conditional entropy $D = H(\text{next} \mid \text{prev})$, a training-free statistic computed before training, we find across controlled corruption families a learning-rate-conditioned law, $\lambda^2 - \lambda_0^2 = C_0(\eta) + C_1(\eta)(H_r - D)^{0.59}$, where $H_r$ is a matched-budget shuffle baseline.

通过使用二元条件熵 $D = H(\text{next} \mid \text{prev})$（这是一种在训练前计算的无需训练的统计量），我们在受控的损坏数据集系列中发现了一个受学习率调节的定律：$\lambda^2 - \lambda_0^2 = C_0(\eta) + C_1(\eta)(H_r - D)^{0.59}$，其中 $H_r$ 是匹配预算的随机基准。

The convex exponent is inherited from an independently measured data-side saturation relation rather than fitted directly to the growth curve. After removing the two per-$\eta$ coefficients, 23 runs spanning an order of magnitude in learning rate collapse onto $(H_r - D)^{0.59}$ with unit slope ($R^2 = 0.941$; direct per-$\eta$ fits are weaker, $R^2 \approx 0.82$).

该凸指数继承自独立测量的数据侧饱和关系，而非直接从增长曲线拟合得出。在移除两个与学习率 $\eta$ 相关的系数后，跨越一个数量级学习率的 23 次运行结果在 $(H_r - D)^{0.59}$ 上呈现出单位斜率的坍缩（$R^2 = 0.941$；直接针对每个 $\eta$ 的拟合效果较弱，$R^2 \approx 0.82$）。

Because $D$ is computed before training, the law is a forward predictor: an end-to-end self-validation recovers held-out within-family weight growth with 5.7% relative error. The readout holds at model and per-layer resolutions and across two tested architectures, with the functional form preserved and only the coefficients changing.

由于 $D$ 是在训练前计算的，该定律具有前向预测能力：端到端的自我验证表明，其对同系列中留存数据的权重增长预测相对误差仅为 5.7%。该结论在模型和层级分辨率上均成立，且在两种测试架构中，函数形式保持不变，仅系数发生变化。

It also marks its boundary: cross-corpus prediction over-predicts code, implicating redundancy as a second axis of a broader $\Phi(D,R,A,H)$ data-to-weight framework.

该研究还指出了其局限性：跨语料库预测在代码数据上会出现过高估计，这暗示了冗余度是更广泛的 $\Phi(D,R,A,H)$ 数据到权重框架中的第二个维度。