Concept-based Visual Counterfactual Explanations with Diffusion Models

基于概念的扩散模型视觉反事实解释

Abstract: Visual counterfactual explanations aim to answer “what minimal change to this image would flip the model’s prediction?”, and are increasingly important as vision models are deployed in safety-critical domains (e.g., medicine). Existing diffusion-based methods can produce realistic edits, but they rely on external classifiers that must work reliably on noisy images, which makes them fragile and hard to deploy for robust explanations.

摘要： 视觉反事实解释旨在回答“对图像进行什么样的最小改动可以改变模型的预测结果？”这一问题。随着视觉模型在医疗等安全关键领域的部署，这类解释变得日益重要。现有的基于扩散模型的方法虽然能生成逼真的编辑效果，但它们依赖于外部的分类器，而这些分类器必须在噪声图像上保持可靠工作，这使得它们在提供稳健解释时显得脆弱且难以部署。

We introduce C-VCE, a new diffusion framework that builds the classifier directly into the generative model via a concept bottleneck layer, so that counterfactuals are guided by human-interpretable features (concepts) instead of a separate noise-robust classifier that works with pixel-level edits. Our model lets users to toggle on/off semantic concepts during sampling, then minimally adjusts relevant image regions, while preserving the rest of the image, respecting feature correlations.

我们引入了 C-VCE，这是一个全新的扩散框架。它通过概念瓶颈层（concept bottleneck layer）将分类器直接构建在生成模型中，从而使反事实生成过程由人类可解释的特征（概念）引导，而非依赖于处理像素级编辑的独立噪声稳健分类器。我们的模型允许用户在采样过程中开启或关闭语义概念，随后对图像的相关区域进行最小程度的调整，同时保留图像的其余部分，并尊重特征之间的相关性。

To keep edits small and controlled, we add a simple probabilistic regularizer that balances “change the prediction” against “stay close to the original”, plus a gradient-based mask that confines modifications to the most relevant regions. On benchmarks such as CelebA, C-VCE matches or improves flip rates while producing counterfactuals that are visually closer to the input and less distorted than baselines that depend on separate noisy-image classifiers.

为了保持编辑的微小且可控，我们添加了一个简单的概率正则化器，用以平衡“改变预测结果”与“保持与原始图像接近”之间的关系，并结合基于梯度的掩码将修改限制在最相关的区域内。在 CelebA 等基准测试中，C-VCE 在达到或超过现有翻转率（flip rates）的同时，生成的反事实图像在视觉上比依赖独立噪声图像分类器的基准方法更接近输入图像，且失真更少。

These properties make C-VCE a practical tool for vision systems where users need concrete “what-if” images without having to trust an additional, noise-robust classifier. More broadly, our results suggest that exposing and controlling an internal concept layer is a promising way to make powerful generative models easier to understand and safer to use.

这些特性使 C-VCE 成为视觉系统的实用工具，用户无需信任额外的噪声稳健分类器，即可获得具体的“假设性（what-if）”图像。从更广泛的角度来看，我们的研究结果表明，暴露并控制内部概念层是使强大的生成模型更易于理解、更安全使用的一种有前景的方法。