Where to Place the Query? Unveiling and Mitigating Positional Bias in In-Context Learning for Diffusion LLMs via Decoding Dynamics
Where to Place the Query? Unveiling and Mitigating Positional Bias in In-Context Learning for Diffusion LLMs via Decoding Dynamics
查询应置于何处？通过解码动力学揭示并缓解扩散大语言模型上下文学习中的位置偏差
While In-Context Learning (ICL) is extensively studied in Autoregressive (AR) LLMs, its mechanism within Diffusion Large Language Models (dLLMs) remains largely unexplored. Unlike AR models restricted by unidirectional causal masking, dLLMs intrinsically utilize bidirectional attention, offering extensive spatial flexibility for query placement.
尽管上下文学习（ICL）在自回归（AR）大语言模型中已得到广泛研究，但其在扩散大语言模型（dLLMs）中的机制仍未被充分探索。与受限于单向因果掩码的自回归模型不同，dLLMs 本质上利用双向注意力机制，为查询放置提供了广泛的空间灵活性。
Unfortunately, current practices conventionally inherit AR-style trailing-query templates, often overlooking the structural paradigm shift. This paper presents a comprehensive analysis unveiling that query position is actually a first-order variable in dLLMs. Through empirical decoupling, we demonstrate that positional variance impacts generation quality on par with example semantic quality.
遗憾的是，目前的实践通常沿用自回归风格的“尾部查询”模板，往往忽视了结构范式的转变。本文通过全面分析揭示，查询位置实际上是 dLLMs 中的一阶变量。通过经验性解耦，我们证明了位置差异对生成质量的影响与示例语义质量相当。
Internally, this positional sensitivity stems from a spatial “Recency Effect” in attention flow and task-dependent shifts in decoding trajectories. To mitigate this instability without ground-truth labels, we reveal that traditional single-step confidence ($C_{decoded}$) fails in dLLMs. Instead, we propose Average Confidence ($\overline{C}$), a novel metric tracking the iterative decoding process.
从内部来看，这种位置敏感性源于注意力流中的空间“近因效应”以及解码轨迹中与任务相关的偏移。为了在没有真实标签的情况下缓解这种不稳定性，我们发现传统的单步置信度（$C_{decoded}$）在 dLLMs 中失效。因此，我们提出平均置信度（$\overline{C}$），这是一种追踪迭代解码过程的新型指标。
By establishing the foundational spatial ICL baselines, we introduce Auto-ICL, a training-free adaptive routing strategy that dynamically optimizes query placement, robustly approaching oracle performance across heterogeneous reasoning and perception tasks.
通过建立基础的空间 ICL 基准，我们引入了 Auto-ICL，这是一种无需训练的自适应路由策略，能够动态优化查询放置，并在各种异构推理和感知任务中稳健地逼近预言机（oracle）性能。