A Mesoscopic View of Transformer Weights Through Row and Column Scale Fields

通过行与列尺度场对 Transformer 权重进行介观视角分析

Abstract: Pooled statistics of Transformer weights obscure how magnitude is distributed across functional channels, while individual weights are too numerous to compare directly. We study the mesoscopic level between them: row and column scale fields, the median-centred log-RMS profiles of a weight matrix over its channels, which together with a global scale and a full balanced core represent the matrix exactly.

摘要： Transformer 权重的汇总统计掩盖了量级在功能通道间的分布方式，而单个权重又过于庞大，难以直接进行比较。我们研究了介于两者之间的介观层面：行与列尺度场（row and column scale fields）。这是权重矩阵在其通道上的中位数中心化对数均方根（log-RMS）分布，它与全局尺度及完全平衡的核心共同构成了矩阵的精确表示。

Across public Pythia checkpoints at four sizes and controlled runs from three initialization families, balancing reveals similar measured core magnitude profiles. A mixture bridge, with its form fixed before the analysis and its coefficients fitted, predicts the pooled-shape departure from field width on held-out runs and data arms of the controlled grid.

通过对四种规模的公开 Pythia 检查点以及来自三个初始化系列的受控运行进行分析，平衡化处理揭示了相似的测量核心量级分布。我们构建了一个混合桥（mixture bridge），其形式在分析前固定，系数通过拟合得出，该模型能够预测受控网格中留出运行（held-out runs）和数据分支上汇总形状偏离场宽的情况。

The indexed fields retain further structure: they align across projections that share a functional channel, and query/key profiles follow reassigned RoPE frequencies rather than fixed matrix coordinates. Training trajectories show early field formation followed by component-dependent broadening or recession.

这些索引场保留了进一步的结构：它们在共享功能通道的投影之间保持一致，且查询/键（query/key）分布遵循重新分配的 RoPE 频率，而非固定的矩阵坐标。训练轨迹显示，场结构在早期形成，随后出现依赖于组件的拓宽或收缩。

Extending the channel-based analysis to AdamW’s second moment reveals related functional organization in its log-space row and column factors. Finally, edits of a frozen checkpoint separate reciprocal scale balance, which preserves the forward computation, from relative channel gain: flattening the gain increases in-distribution loss while preserving matrix norms and the balanced core. Row and column scale fields thus connect pooled magnitude statistics to channel organization and provide coordinates for tracking and testing trained weight structure.

将基于通道的分析扩展到 AdamW 的二阶矩，揭示了其对数空间行与列因子中相关的函数组织结构。最后，通过对冻结检查点的编辑，我们将保持前向计算的倒数尺度平衡与相对通道增益分离开来：平滑增益会增加分布内损失，同时保持矩阵范数和平衡核心不变。因此，行与列尺度场将汇总的量级统计与通道组织联系起来，并为跟踪和测试已训练的权重结构提供了坐标。