Counterfactual Neighborhood Risk Fields: Evidence-Grounded Selective Safety for Vision-Language Models

Representation-based safety defenses for vision-language models (VLMs) often learn to separate harmful from benign inputs, but separation alone does not establish what a risk score means or when it should be trusted.

基于表征的视觉语言模型（VLM）安全防御通常学习如何区分有害输入和良性输入，但仅靠区分并不能确定风险评分的含义，也无法确定何时该信任这些评分。

We introduce Counterfactual Neighborhood Risk Fields (CNRF), an evidence-grounded framework for selective VLM safety. CNRF constructs context-matched benign-harmful counterfactual pairs, uses their representation differences as capability-related risk directions, and estimates neighborhood support separately from risk.

我们引入了反事实邻域风险场（CNRF），这是一个用于选择性 VLM 安全的证据基础框架。CNRF 构建上下文匹配的良性-有害反事实对，利用它们的表征差异作为与能力相关的风险方向，并独立于风险来估计邻域支持度。

This separation prevents unfamiliarity from being conflated with harmfulness and enables selective intervention rather than blanket rejection. A victim-adaptive interface accommodates different safety observations across VLM architectures and routes selected requests for same-victim response review, which either releases acceptable candidates or produces constrained alternatives with reduced harmful capability.

这种分离防止了将陌生感与有害性混为一谈，并实现了选择性干预而非一刀切的拒绝。一个受害者自适应接口能够适应不同 VLM 架构下的安全观察，并将选定的请求路由至同受害者响应审查，从而发布可接受的候选结果或生成降低了有害能力的受限替代方案。

Across three VLMs and five multimodal jailbreak families, CNRF achieves 86.06-89.30% macro defense success rate while retaining substantial benign-task utility. Controlled experiments further show that query-matched references and preserved context-direction correspondence materially improve risk ranking in the low-false-positive regime.

在三个 VLM 和五个多模态越狱家族的测试中，CNRF 实现了 86.06-89.30% 的宏观防御成功率，同时保留了显著的良性任务效用。对照实验进一步表明，查询匹配的参考资料和保留的上下文方向对应关系在低误报率环境下显著改善了风险排序。

CNRF thus reframes representation-based VLM safety from learning harmfulness boundaries to constructing applicable evidence for selective risk decisions.

因此，CNRF 将基于表征的 VLM 安全性从学习有害性边界重新定义为构建用于选择性风险决策的适用证据。