When Irrelevant Text Matters: Affine Margin Shifts in Multimodal Large Language Models

当无关文本产生影响：多模态大语言模型中的仿射边界偏移

Abstract: Multimodal large language models (MLLMs) are frequently exposed to auxiliary textual context, the impact of which on visually grounded tasks remains underexplored. 摘要： 多模态大语言模型（MLLMs）经常接触到辅助性文本上下文，但这些上下文对视觉基础任务的影响尚缺乏深入研究。

In this paper, we investigate the influence of task-irrelevant context by formulating it as a controlled intervention within a binary visual judgment framework. By maintaining an invariant prompt structure while varying auxiliary inputs, we observe that irrelevant text consistently biases model predictions across diverse benchmarks. 在本文中，我们通过在二元视觉判断框架内将其设定为受控干预，研究了任务无关上下文的影响。通过保持提示词结构不变并改变辅助输入，我们观察到无关文本在多个基准测试中持续地对模型预测产生偏差。

To move beyond performance metrics, we characterize this sensitivity through a decision margin defined by the log-probability difference between binary candidates. Our analysis reveals a robust geometric regularity: context-conditioned margins follow a consistent affine transformation of their context-free counterparts. 为了超越单纯的性能指标，我们通过由二元候选词之间的对数概率差定义的“决策边界”来刻画这种敏感性。我们的分析揭示了一种稳健的几何规律：受上下文影响的边界遵循其无上下文对应项的一致仿射变换。

This finding demonstrates that irrelevant context does not manifest as unstructured stochastic noise but as a estimable distortion of model preference. We further interpret the fitted affine parameters as metrics for visual commitment preservation and directional answer bias. 这一发现表明，无关上下文并非表现为无结构的随机噪声，而是表现为模型偏好的可估计畸变。我们进一步将拟合出的仿射参数解释为衡量视觉承诺保持度和方向性答案偏差的指标。

These findings provide a margin-level diagnostic view of irrelevant-context effects in MLLMs and offer a basis for future studies on noisy-context robustness. 这些发现为多模态大语言模型中无关上下文效应提供了边界层面的诊断视角，并为未来关于噪声上下文鲁棒性的研究奠定了基础。