Mitigating Bias in Large Vision-Language Models via Counterfactual Ensemble Decoding
Mitigating Bias in Large Vision-Language Models via Counterfactual Ensemble Decoding
通过反事实集成解码缓解大型视觉语言模型中的偏见
Abstract: Large Vision-Language Models (LVLMs) have achieved remarkable performance across a wide range of tasks; however, they often inherit social biases from their training data, resulting in biased behavior when processing portraits from different social groups. 摘要： 大型视觉语言模型（LVLMs）在广泛的任务中取得了卓越的性能；然而，它们往往会从训练数据中继承社会偏见，导致在处理来自不同社会群体的肖像时表现出偏见行为。
Existing debiasing approaches typically compare token probabilities between the original and biased generations during decoding, but they are fundamentally limited by their reliance on a single, stereotyped viewpoint and fail to account for the diversity of social perspectives. 现有的去偏方法通常在解码过程中比较原始生成与偏见生成之间的标记（token）概率，但它们从根本上受到限制，因为它们依赖于单一的刻板印象视角，且未能考虑到社会视角的多元性。
Inspired by the social science principle that diversity fosters fairness, we propose Counterfactual Ensemble Decoding (CED), a novel framework that constructs multi-group counterfactual perspectives within the visual representation space and integrates them during decoding to promote equitable model behavior. 受“多样性促进公平”这一社会科学原则的启发，我们提出了反事实集成解码（Counterfactual Ensemble Decoding, CED）。这是一个新颖的框架，它在视觉表征空间内构建多群体反事实视角，并在解码过程中对其进行集成，以促进模型行为的公平性。
CED first performs counterfactual steering in the visual space by identifying semantic directions associated with each social group and generating counterfactual representations along these directions, thereby offering diverse perspectives that disrupt stereotypical narratives. CED 首先通过识别与每个社会群体相关的语义方向，并在这些方向上生成反事实表征，从而在视觉空间中执行反事实引导，进而提供能够打破刻板叙事的多元视角。
During decoding, CED locates the decoder layer exhibiting the greatest divergence among these perspectives and ensembles their token distributions using uncertainty-aware weights, prioritizing high-confidence tokens from different groups to yield a more balanced probability distribution that guides fairer generation. 在解码过程中，CED 定位出在这些视角间表现出最大差异的解码器层，并使用不确定性感知权重对它们的标记分布进行集成，优先考虑来自不同群体的高置信度标记，从而产生更平衡的概率分布，以引导更公平的生成。
Extensive experiments on three social bias evaluation benchmarks demonstrate that CED achieves substantial improvements over leading baselines, reducing bias by up to 47.97% across scenarios involving occupations, descriptors, and persona traits. Moreover, CED also preserves the core capabilities of the original model with minimal degradation. 在三个社会偏见评估基准上的大量实验表明，CED 在职业、描述词和个人特质等场景中，较主流基准模型取得了显著改进，偏见降低幅度高达 47.97%。此外，CED 在保持原始模型核心能力的同时，性能损耗极小。