Asymmetric Attention Heads: Structured Head-Wise Context Allocation for Transformer Attention

非对称注意力头：Transformer 注意力中结构化的逐头上下文分配

Standard multi-head attention (MHA) gives every head the same full causal context span, although heads can serve different contextual roles. Some heads may rely mainly on nearby lexical or syntactic context, while others may depend on longer-range relations such as entity interactions, discourse links, or state changes.

标准的多头注意力（MHA）机制为每个注意力头分配相同的完整因果上下文跨度，尽管各个头在实际应用中可能承担不同的上下文角色。某些头可能主要依赖于近距离的词法或句法上下文，而另一些头则可能依赖于更长距离的关系，例如实体交互、语篇链接或状态变化。

We present Asymmetric Attention Heads (AAH), a head-wise context-allocation framework that treats context length as an explicit per-head or per-group allocation variable. AAH groups heads using feature-derived statistics, organizes these groups hierarchically, and assigns causal local windows while preserving the standard flat MHA output interface.

我们提出了非对称注意力头（Asymmetric Attention Heads, AAH），这是一个逐头的上下文分配框架，它将上下文长度视为一个显式的、针对每个头或每个组的分配变量。AAH 利用特征导出的统计数据对注意力头进行分组，以分层方式组织这些组，并在保持标准扁平化 MHA 输出接口的同时，分配因果局部窗口。

In 4096-token seed-0 experiments, several AAH-style local-allocation variants achieve lower validation loss than pure full attention. Short-budget ablations show that stable local allocation and head-window assignment structure matter, while fixed/local controls can be competitive with adaptive hierarchy.

在 4096 个 token 的 seed-0 实验中，几种 AAH 风格的局部分配变体实现了比纯全注意力机制更低的验证损失。短预算消融实验表明，稳定的局部分配和头窗口分配结构至关重要，而固定的/局部的控制方法在性能上可以与自适应分层方法相媲美。

We interpret AAH as a structured head-wise context-allocation mechanism for quality and analysis, with Attention Coverage Ratio (ACR) reported as a selected-window routing diagnostic.

我们将 AAH 解释为一种用于质量评估和分析的结构化逐头上下文分配机制，并报告了注意力覆盖率（Attention Coverage Ratio, ACR）作为选定窗口路由的诊断指标。