It Takes Little to Rewrite Perception: Targeted Semantic Substitution in Vision-Language Models at $\epsilon \leq 4/255$

只需微小扰动即可重写感知：视觉语言模型在 $\epsilon \leq 4/255$ 下的目标语义替换

Abstract: Vision Language Models (VLMs) are widely deployed in safety-critical scenarios, and understanding to which extent they can be controlled by adversarial perturbation is a prerequisite for evaluating their trustworthiness. Existing representation-alignment attacks, which make a VLM perceive a target image, achieve limited success at $\varepsilon \leq 4/255$. Therefore, VLMs seems robust to perturbations in this range.

摘要： 视觉语言模型（VLMs）已被广泛部署于安全关键场景中，了解它们在多大程度上能被对抗性扰动所控制，是评估其可信度的前提。现有的表征对齐攻击旨在使 VLM 将输入图像感知为目标图像，但在 $\varepsilon \leq 4/255$ 的扰动范围内成功率有限。因此，VLM 在此范围内似乎表现出较强的鲁棒性。

We show that this robustness does not hold, as targeted semantic substitution succeeds within the same range. Specifically, we align each stream of the source image with its counterpart in the target image in the victim VLM’s post-merger token space, operating under a white-box threat model. We evaluate under a strict success criterion, requiring the model to simultaneously name the target, confirm its presence, and deny the source.

我们证明这种鲁棒性并不成立，因为目标语义替换在相同的扰动范围内可以成功实现。具体而言，我们在白盒威胁模型下，将源图像的每个流与受害者 VLM 合并后 Token 空间中的目标图像对应部分进行对齐。我们采用了严格的成功标准进行评估，要求模型同时能够命名目标、确认其存在，并否认源图像的存在。

In images, target semantics appear at $\varepsilon = 2/255$ and complete replacement reaches 38% at $\varepsilon = 4/255$. On video, complete replacement reaches 35.9% at $\varepsilon = 1/255$. We also observe a phenomenon of \textit{semantic fusion}, where Large Language Model (LLM) rationalizes contradictory visual signals into a coherent narrative.

在图像任务中，目标语义在 $\varepsilon = 2/255$ 时开始显现，而在 $\varepsilon = 4/255$ 时完全替换率达到 38%。在视频任务中，完全替换率在 $\varepsilon = 1/255$ 时达到 35.9%。我们还观察到一种“语义融合”现象，即大语言模型（LLM）会将相互矛盾的视觉信号合理化为连贯的叙述。