TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs

TokenSwap：基准测试并缩小多模态大语言模型中的模态差距

Abstract: Multimodal large language models (MLLMs) should generate consistent responses given semantically equivalent inputs across modalities. However, we observe a systematic discrepancy in model predictions under such cross-modal variations. Specifically, we define the modality gap as the difference in model performance under semantically equivalent textual and multimodal inputs.

摘要： 多模态大语言模型（MLLMs）在面对跨模态的语义等价输入时，理应生成一致的响应。然而，我们观察到模型在处理此类跨模态变化时，预测结果存在系统性的差异。具体而言，我们将“模态差距”（modality gap）定义为模型在处理语义等价的文本输入与多模态输入时，其性能表现上的差异。

We introduce TokenSwap, a method that constructs such inputs by replacing textual concepts with semantically aligned images, resulting in sequences where visual tokens are interleaved with text tokens. Based on TokenSwap, we transform existing text-based benchmarks such as MMLU into image-interleaved counterparts, resulting in TokenSwap-Bench.

我们引入了 TokenSwap，这是一种通过将文本概念替换为语义对齐的图像来构建此类输入的方法，从而生成视觉标记与文本标记交错的序列。基于 TokenSwap，我们将 MMLU 等现有的文本基准测试转换为图像交错的对应版本，从而构建了 TokenSwap-Bench。

Across 42 MLLMs, we observe a pervasive modality gap, with performance decreasing by 4.2% to 47.4% when moving from text-only to image-interleaved inputs, averaging 19.6% +/- 3.3% across models. Notably, we observe that reasoning models exhibit consistently smaller gaps, achieving an average gap of 10.1% compared to 25.5% for non-reasoning models.

在对 42 个 MLLM 的测试中，我们观察到普遍存在的模态差距：当输入从纯文本转换为图像交错格式时，模型性能下降了 4.2% 到 47.4%，平均下降幅度为 19.6% +/- 3.3%。值得注意的是，我们发现推理模型表现出的差距始终较小，其平均差距为 10.1%，而普通模型则为 25.5%。

In contrast, neither prompting strategies nor scaling training compute alone reliably reduces the modality gap. Finally, we demonstrate that incorporating TokenSwap during training effectively mitigates this gap while preserving strong text-only and vision-language performance.

相比之下，无论是提示工程策略还是单纯增加训练计算量，都无法可靠地缩小模态差距。最后，我们证明了在训练过程中引入 TokenSwap 可以有效缓解这一差距，同时保持模型在纯文本和视觉语言任务上的强大性能。