WavePrune: One period is often enough for RoPE

Computer Science > Computation and Language arXiv:2610.06963 (cs) [Submitted on 3 Oct 2026] Title: WavePrune: One period is often enough for RoPE Authors: Guancheng Du, Luotian Huang, Shaowen Wang, Si Li, Kaifeng Lyu.

计算机科学 > 计算与语言 arXiv:2610.06963 (cs) [提交于 2026 年 10 月 3 日] 标题：WavePrune：对于 RoPE 而言，一个周期通常就足够了 作者：Guancheng Du, Luotian Huang, Shaowen Wang, Si Li, Kaifeng Lyu。

Abstract: Rotary Position Embedding (RoPE) encodes token positions by rotating each two-dimensional channel of the query and key vectors at a channel-specific frequency, making the attention logits invariant to a common shift of positions. However, this rotation is periodic, and it leads to position aliasing where relative positions separated by a full rotation period become hard to tell apart.

摘要：旋转位置编码（RoPE）通过以特定通道频率旋转查询向量和键向量的每个二维通道来编码标记位置，使得注意力逻辑值对于位置的共同偏移保持不变。然而，这种旋转是周期性的，这会导致位置混叠，使得相隔一个完整旋转周期的相对位置难以区分。

To address this, we propose WavePrune, which restricts each channel to its first rotation period. We show that it removes the distractions in attention maps created by position aliasing and improves overall long-context performance. Specifically, WavePrune raises the HELMET score on four of five models we test without any extra tuning (e.g., 35.7 -> 40.0 on Qwen3-8B).

为了解决这个问题，我们提出了 WavePrune，它将每个通道限制在其第一个旋转周期内。我们证明了这消除了由位置混叠在注意力图中产生的干扰，并提高了整体的长上下文性能。具体而言，WavePrune 在我们测试的五个模型中的四个上提高了 HELMET 分数，且无需任何额外调整（例如，在 Qwen3-8B 上从 35.7 提高到 40.0）。

When pretraining models from scratch, WavePrune also achieves lower validation loss at extrapolated lengths than pretraining without it. Because WavePrune restricts each channel to a sliding window, it induces a fine-grained sparsity that our hardware-aligned CUDA kernels exploit for 1.15x prefill and 1.24x decoding speedups over FlashAttention-2 at 32K context.

在从头开始预训练模型时，WavePrune 在外推长度上也比不使用它进行预训练实现了更低的验证损失。由于 WavePrune 将每个通道限制在一个滑动窗口内，它引入了一种细粒度的稀疏性，我们的硬件对齐 CUDA 内核利用这种稀疏性，在 32K 上下文下实现了比 FlashAttention-2 快 1.15 倍的预填充速度和 1.24 倍的解码速度。

Together, these results show that RoPE’s periodic structure, widely regarded as essential, is largely redundant beyond the first rotation period.

总之，这些结果表明，被广泛认为必不可少的 RoPE 周期性结构在第一个旋转周期之后很大程度上是多余的。