Position Encoding in Transformers: From Absolute and Relative Methods to Rotary Position Embeddings and Long-Context Scaling
Position Encoding in Transformers: From Absolute and Relative Methods to Rotary Position Embeddings and Long-Context Scaling
Transformer 中的位置编码：从绝对与相对方法到旋转位置嵌入及长上下文扩展
Abstract: Self-attention models content-dependent interactions between tokens but does not by itself encode token order. Position encoding addresses this limitation by introducing absolute coordinates, relative distances, or position-dependent rotations into Transformer representations and attention scores.
摘要： 自注意力机制能够建模 Token 之间依赖于内容（content-dependent）的交互，但其本身并不能编码 Token 的顺序。位置编码通过在 Transformer 表示和注意力分数中引入绝对坐标、相对距离或与位置相关的旋转，解决了这一局限性。
This technical survey develops a unified account of sinusoidal and learned absolute position embeddings, Shaw-style relative position representations, Transformer-XL, T5 relative position bias, ALiBi, and Rotary Position Embeddings (RoPE). We derive how RoPE converts absolute position indices into relative phase differences in Query-Key inner products and compare these methods in terms of where position is injected, computational cost, compatibility with KV caching, and length extrapolation.
本技术综述对正弦式与学习式绝对位置嵌入、Shaw 风格的相对位置表示、Transformer-XL、T5 相对位置偏置、ALiBi 以及旋转位置嵌入（RoPE）进行了统一的阐述。我们推导了 RoPE 如何将绝对位置索引转换为 Query-Key 内积中的相对相位差，并从位置注入方式、计算成本、KV 缓存兼容性以及长度外推性等方面对这些方法进行了比较。
We then examine long-context extensions, including Position Interpolation, RoPE scaling laws, NTK-aware scaling, Dynamic NTK, NTK-by-parts, YaRN, LongRoPE, and LongRoPE2, with emphasis on frequency allocation, attention rescaling, training length, and target context length. We also summarize implementation considerations, evaluation protocols, and position-encoding choices in representative large language models.
随后，我们探讨了长上下文扩展技术，包括位置插值（Position Interpolation）、RoPE 缩放定律、NTK 感知缩放、动态 NTK、分段 NTK（NTK-by-parts）、YaRN、LongRoPE 和 LongRoPE2，重点关注频率分配、注意力重缩放、训练长度和目标上下文长度。我们还总结了代表性大语言模型中的实现考量、评估协议以及位置编码的选择。
A central conclusion is that the ability to compute positional features beyond the training length does not imply reliable long-context generalization; context extension must be evaluated through short-context retention, position-wise perplexity, retrieval, reasoning, and long-context code tasks.
一个核心结论是：计算超出训练长度的位置特征的能力，并不意味着具备可靠的长上下文泛化能力；上下文扩展必须通过短上下文保留能力、位置困惑度（position-wise perplexity）、检索、推理以及长上下文代码任务来进行评估。