WAV: Multi-Resolution Block Residual Routing for Deep Decoder-Only Transformers

WAV：面向深度仅解码器 Transformer 的多分辨率块残差路由

Abstract: Residual connections are central to training deep Transformers, but standard PreNorm residual streams aggregate sublayer updates with fixed unit weights. Recent Attention Residuals replace this fixed accumulation with content-dependent depth-wise routing, and Block Attention Residuals make the mechanism efficient by routing over block-level residual summaries. However, a single block summary stores only the low-frequency total residual displacement inside a block, discarding directional structure such as attention-vs-MLP imbalance and early-vs-late block dynamics.

摘要： 残差连接是训练深度 Transformer 的核心，但标准的 PreNorm 残差流通过固定的单位权重来聚合子层更新。近期的注意力残差（Attention Residuals）用内容相关的深度路由取代了这种固定累加，而块注意力残差（Block Attention Residuals）则通过对块级残差摘要进行路由，提高了该机制的效率。然而，单一的块摘要仅存储了块内低频的总残差位移，丢失了诸如注意力与 MLP 之间的不平衡以及块内早期与晚期动态等方向性结构。

We propose WAV v1, a lightweight multi-resolution residual routing method for decoder-only Transformers. Instead of representing each block only by its accumulated residual sum, WAV v1 augments every block with two directional detail bases: a phase basis that contrasts attention and MLP updates, and a split basis that contrasts early and late sublayer updates. These bases are routed together with standard block summaries through the same depth-wise softmax mixer, while negative detail-source initialization and detached RMS matching stabilize training.

我们提出了 WAV v1，这是一种面向仅解码器 Transformer 的轻量级多分辨率残差路由方法。WAV v1 不再仅用累积残差和来表示每个块，而是为每个块增加了两个方向性细节基：一个是对比注意力与 MLP 更新的“相位基”（phase basis），另一个是对比早期与晚期子层更新的“分割基”（split basis）。这些基与标准的块摘要一起，通过相同的深度 Softmax 混合器进行路由，同时利用负细节源初始化和分离的 RMS 匹配来稳定训练过程。

On character-level TinyStories and Text8 language modeling, WAV v1 shows a clear depth-dependent benefit. Although it is not consistently beneficial at 12 layers, it becomes competitive at 24 layers and outperforms all baselines at 48 layers. At 48 layers, WAV v1 reduces validation loss relative to Block AttnRes from 0.4960 to 0.4738 on TinyStories and from 0.9363 to 0.9305 on Text8, with negligible additional parameters. These results suggest that directional residual details, not only block-level sums, are important for scaling residual routing in deeper Transformers.

在字符级 TinyStories 和 Text8 语言建模任务中，WAV v1 显示出明显的深度依赖优势。尽管在 12 层时其优势并不显著，但在 24 层时已具备竞争力，并在 48 层时超越了所有基准模型。在 48 层配置下，相比于块注意力残差（Block AttnRes），WAV v1 将 TinyStories 上的验证损失从 0.4960 降低至 0.4738，将 Text8 上的验证损失从 0.9363 降低至 0.9305，且仅增加了可忽略不计的参数量。这些结果表明，对于扩展更深层 Transformer 中的残差路由而言，方向性残差细节与块级总和同样重要。