Compliance, Capability, and Conflict: Benchmarking Multimodal LLMs under System Messages

合规性、能力与冲突：系统提示词下多模态大模型的基准测试

Abstract: Production deployments of Multimodal Large Language Models (MLLMs) increasingly rely on system messages to govern model behavior. Yet existing benchmarks either evaluate constraints in text only or embed them into the user turn, leaving system-message adherence in multimodal contexts largely unmeasured; they also leave open whether compliance comes at the cost of foundational vision-language capabilities.

摘要： 多模态大模型（MLLM）的生产部署日益依赖系统提示词（System Messages）来规范模型行为。然而，现有的基准测试要么仅评估纯文本约束，要么将约束嵌入到用户指令中，导致多模态场景下模型对系统提示词的遵循情况在很大程度上未得到衡量；此外，这些测试也未明确模型在追求合规性的同时，是否会以牺牲基础视觉-语言能力为代价。

We introduce VSysBench, a benchmark built on MMVet-v2 that organizes constraints into 5 main categories and 22 sub-categories, ranging from textual directives in visual contexts to fully vision-grounded ones, each paired with a misaligned counterpart that stress-tests the instructional hierarchy.

我们推出了 VSysBench，这是一个基于 MMVet-v2 构建的基准测试。它将约束条件划分为 5 个主要类别和 22 个子类别，涵盖了从视觉上下文中的文本指令到完全基于视觉的指令，并为每一项都配对了冲突的对照组，以压力测试模型的指令层级处理能力。

VSysBench scores each response jointly along two axes, constraint compliance and answer correctness, via the Joint Satisfaction Rate (JSR) and Cross-Constraint Sensitivity (CCS).

VSysBench 通过联合满足率（JSR）和跨约束敏感度（CCS）两个维度，对每个回复进行联合评分，分别衡量其约束合规性和答案准确性。

Across 16 MLLMs, we find that imposing system messages substantially erodes base task accuracy, that compliance collapses under user conflict for open-weight models while remaining stable for top proprietary ones, and that vision-grounded constraints are the hardest category for every model.

通过对 16 个多模态大模型的测试，我们发现：施加系统提示词会显著降低模型的基础任务准确率；在面对用户指令冲突时，开源权重模型的合规性会崩溃，而顶级闭源模型则保持稳定；此外，对于所有模型而言，基于视觉的约束都是最具挑战性的类别。