Complexity-Guided Component-wise Initialization for Language Model Pretraining

基于复杂性引导的语言模型预训练组件级初始化

Abstract: Pretrained language models often exhibit structured weight spectra, suggesting that training may repeatedly produce similar layerwise and component-wise organization. We ask whether these recurring spectral patterns can be reused as an initialization signal for GPT-2-style language-model pretraining.

摘要： 预训练语言模型通常表现出结构化的权重谱，这表明训练过程可能会反复产生相似的层级和组件级组织结构。我们探讨了这些反复出现的谱模式是否可以作为 GPT-2 风格语言模型预训练的初始化信号进行复用。

First, we analyze eleven pretrained GPT-2-style checkpoints that vary in size, language, tokenizer, and training corpus, measuring Frobenius norm and effective-rank entropy across layers and Transformer subcomponents. The checkpoints show shared depth trends, especially increasing scale and stronger spectral concentration in residual-writing matrices.

首先，我们分析了十一个在规模、语言、分词器和训练语料库上各不相同的预训练 GPT-2 风格检查点，并测量了各层及 Transformer 子组件的 Frobenius 范数和有效秩熵。这些检查点显示出共同的深度趋势，特别是在残差写入矩阵中表现出的规模增加和更强的谱集中度。

We then construct initialization schemes that imitate the component-wise magnitudes and spectral profiles of pretrained models, and compare them with several weight initialization methods. These initializers visibly change the model’s structural spectral patterns, but the evaluation results do not show a corresponding performance advantage.

随后，我们构建了模仿预训练模型组件级幅度和谱配置文件的初始化方案，并将其与几种权重初始化方法进行了比较。这些初始化器明显改变了模型的结构谱模式，但评估结果并未显示出相应的性能优势。

Pretrained-weight reuse remains competitive, while coarse spectral matching alone is not a reliable optimization strategy. Our results suggest that pretrained spectra are useful diagnostics of trained model structure, but that effective reuse likely requires preserving richer information than component-wise scale and singular-value shape.

预训练权重的复用仍然具有竞争力，但仅靠粗略的谱匹配并不是一种可靠的优化策略。我们的研究结果表明，预训练谱是诊断已训练模型结构的有用工具，但有效的复用可能需要保留比组件级规模和奇异值形状更丰富的信息。