The Structural Attention Tax: How Retrieval Format Hijacks In-Context Learning Independent of Content

结构性注意力税：检索格式如何独立于内容劫持上下文学习

Abstract: Retrieval-augmented generation (RAG) systems inject external knowledge to improve LLM outputs, yet the format of injected content — distinct from its semantic relevance — can independently distort the model’s attention distribution. We identify and formalise a phenomenon we term the structural attention tax: knowledge graph (KG) triples, due to their relational delimiters and repeated slot patterns, capture 2-3x more attention per token than semantically equivalent natural-language text ($\hat{o}$(KG) $\approx$ 0.70 vs. $\hat{o}$(neutral) $\approx$ 0.25), compressing demonstration attention by up to 42% — regardless of whether the triples are relevant or noise.

摘要： 检索增强生成（RAG）系统通过注入外部知识来改善大语言模型（LLM）的输出，然而注入内容的格式——与其语义相关性无关——可能会独立地扭曲模型的注意力分布。我们识别并形式化了一种被称为“结构性注意力税”的现象：知识图谱（KG）三元组由于其关系分隔符和重复的槽位模式，每个 token 捕获的注意力是语义等效自然语言文本的 2-3 倍（$\hat{o}$(KG) $\approx$ 0.70 vs. $\hat{o}$(neutral) $\approx$ 0.25），这导致演示（demonstration）注意力被压缩了高达 42%——无论这些三元组是相关的还是噪声。

We develop a formal framework decomposing attention scores into semantic and structural components (Eq. 2), derive a compression bound (Proposition 1) connecting token-level format bias to demonstration attention loss, and show that the structural term governs how much attention is diverted while the semantic term governs whether this helps or hurts. This decoupling reveals two orthogonal axes for improving retrieval-augmented ICL: optimising retrieval quality (semantic axis) and reducing format-driven attention capture (structural axis).

我们开发了一个将注意力分数分解为语义和结构成分的形式化框架（公式 2），推导出了一个连接 token 级格式偏差与演示注意力损失的压缩界限（命题 1），并证明了结构项决定了注意力被转移的程度，而语义项则决定了这种转移是有益还是有害。这种解耦揭示了改进检索增强上下文学习（ICL）的两个正交轴：优化检索质量（语义轴）和减少格式驱动的注意力捕获（结构轴）。

Empirically, across two model families (Mistral-7B, LLaMA-3-8B) and three QA benchmarks, we observe that source-task alignment dominates: task-matched BM25 retrieval achieves 58-62% on HotpotQA vs. ConceptNet’s 25-27%, a >30 pp gap that dwarfs all gating strategies ($\leq$2 pp). We derive five structure-aware mitigation strategies from the framework, ranging from zero-cost prompt modifications to training-time regularisation; format flattening (S3) is validated by both accuracy and attention-level evidence from a verbalized-triple control, while structural dispersal (S1) yields mixed results that illuminate the challenges of format-level intervention.

在实证方面，通过两个模型家族（Mistral-7B, LLaMA-3-8B）和三个问答基准测试，我们观察到源任务对齐（source-task alignment）起主导作用：任务匹配的 BM25 检索在 HotpotQA 上达到了 58-62% 的准确率，而 ConceptNet 仅为 25-27%，超过 30 个百分点的差距使所有门控策略（$\leq$2 个百分点）相形见绌。我们从该框架中推导出了五种结构感知缓解策略，范围从零成本的提示词修改到训练时的正则化；格式扁平化（S3）通过准确率和来自口语化三元组对照组的注意力水平证据得到了验证，而结构分散化（S1）的结果则喜忧参半，这凸显了格式级干预所面临的挑战。