Riemannian Geometry for Pre-trained Language Model Embeddings

预训练语言模型嵌入的黎曼几何研究

Abstract: Understanding the geometric structure of pre-trained language model embeddings matters for interpretability and safety. We ask whether sentence-level classification signal lives in the Riemannian geometry of contextual token embeddings, and probe it by extracting per-token pullback metrics from a learned encoder’s analytical Jacobian and aggregating them with the Fréchet mean on the symmetric positive definite (SPD) manifold; we call this procedure Riemannian Mean Pooling (RMP).

摘要： 理解预训练语言模型嵌入的几何结构对于模型的可解释性和安全性至关重要。我们探讨了句子级分类信号是否蕴含在上下文标记（token）嵌入的黎曼几何中，并通过从已学习编码器的解析雅可比矩阵（analytical Jacobian）中提取每个标记的拉回度量（pullback metrics），并利用对称正定（SPD）流形上的弗雷歇均值（Fréchet mean）进行聚合来验证这一假设；我们将此过程称为黎曼均值池化（Riemannian Mean Pooling, RMP）。

Across three datasets with non-trivial linguistic structure (CoLA, CREAK, RTE), RMP outperforms Euclidean mean pooling, while on FEVER-Symmetric, a benchmark constructed to remove annotation-driven lexical artifacts, the method correctly stays at chance.

在三个具有复杂语言结构的数据集（CoLA、CREAK、RTE）上，RMP 的表现优于欧几里得均值池化；而在旨在消除标注驱动的词汇伪影（lexical artifacts）的基准测试 FEVER-Symmetric 上，该方法准确地保持在随机水平。

Ablations show that a randomly initialised encoder combined with Fréchet aggregation already beats Euclidean pooling on two of the three signal-bearing datasets, localising the source of the gain to the geometric aggregation rather than to learned manifold structure; the trained encoder contributes additional signal specifically on CREAK, the most knowledge-heavy of the three signal-bearing datasets.

消融实验表明，随机初始化的编码器结合弗雷歇聚合，在三个包含信号的数据集中的两个上已经优于欧几里得池化，这说明性能提升的来源在于几何聚合而非学习到的流形结构；而经过训练的编码器仅在 CREAK（三个数据集知识密度最高的一个）上贡献了额外的信号。