Hierarchical Grading in Large Language Models

大型语言模型中的分级（Hierarchical Grading）

Abstract: We introduce Graded Large Language Models (GLLMs), an algebraic framework that equips the representation space of a transformer with a grading and propagates the induced weighted scalar action through embeddings, self-attention, and the training objective. The construction extends the theory of graded neural networks and graded transformers to autoregressive language models while preserving expressive power, asymptotic computational complexity, and inference cost.

摘要： 我们引入了分级大型语言模型（Graded Large Language Models, GLLMs），这是一种代数框架，它为 Transformer 的表示空间配备了分级（grading），并将诱导的加权标量作用传播到嵌入层、自注意力机制以及训练目标中。该构建将分级神经网络和分级 Transformer 的理论扩展到了自回归语言模型，同时保持了原有的表达能力、渐近计算复杂度和推理成本。

The governing geometric picture is that of geometric invariant theory. The benefit of a grading is expressed by a Kempf—Ness functional on the grading torus; the grades that improve upon the uniform architecture form an open convex cone whose membership is decided by a Hilbert—Mumford-type criterion pairing a grade direction against two measurable profiles of the target and the data; the optimal grades are the coincidence point of two moment maps, given in closed form; and the ordinary transformer appears as a semistable isotropic point on the boundary of the cone: one member of a larger graded family rather than a distinguished optimum.

其核心几何图景基于几何不变量理论（Geometric Invariant Theory）。分级的优势通过分级环面上的 Kempf—Ness 泛函来体现；优于统一架构的分级构成了一个开凸锥，其成员资格由 Hilbert—Mumford 型准则决定，该准则将分级方向与目标和数据的两个可测分布进行配对；最优分级是两个矩映射（moment maps）的重合点，并以闭式形式给出；而普通 Transformer 则表现为该锥边界上的半稳定各向同性点：它只是一个更大的分级家族中的一员，而非一个特殊的极值点。

Separately, for level-stratified targets we prove a minimax separation between the graded prior and its absence: over all estimators the risks of the graded and uniform target classes separate throughout an explicit window of sample sizes, by a factor that decays exponentially in the number of levels under geometric stratification. Both profiles are estimable offline, so the optimal grades solve a convex program certified before training begins. Because the grading is absorbed into the learned parameters after training, every GLLM compiles to a standard transformer of identical architecture and inference complexity.

此外，针对分层目标，我们证明了分级先验与非分级先验之间的极小极大分离（minimax separation）：在所有估计器中，分级目标类与统一目标类的风险在明确的样本量窗口内存在差异，且在几何分层下，该差异因子随层数的增加呈指数级衰减。这两个分布均可离线估计，因此最优分级可以通过在训练前验证的凸优化程序来求解。由于分级在训练后被吸收到学习参数中，每个 GLLM 最终都会编译为一个具有相同架构和推理复杂度的标准 Transformer。