Measuring Fairness in Large Audio Language Models via Semantic-Aware Bias Estimation

通过语义感知偏差估计衡量大型音频语言模型的公平性

Large Audio Language Models (LALMs) have seen increasing use for audio understanding tasks such as speech recognition and audio question answering, raising concerns about fairness across demographic subgroups. 大型音频语言模型（LALMs）在语音识别和音频问答等音频理解任务中的应用日益广泛，这引发了人们对不同人口统计学子群体之间公平性的担忧。

Fairness evaluation in spoken-input settings is challenging due to confounding factors, including semantic variation in spoken content and speaker-specific characteristics. Ignoring these factors can result in misleading conclusions about model bias. 由于存在混杂因素（包括口语内容的语义差异和说话人特定特征），在语音输入环境下进行公平性评估极具挑战性。忽略这些因素可能会导致对模型偏差得出误导性的结论。

We propose a semantic-aware mixed-effects regression framework for fairness evaluation in LALMs that explicitly accounts for these confounders. Our approach incorporates sentence-level semantic embeddings of reference text as covariates and models speaker identity as a random effect. 我们提出了一种用于 LALM 公平性评估的语义感知混合效应回归框架，该框架明确考虑了这些混杂因素。我们的方法将参考文本的句子级语义嵌入作为协变量，并将说话人身份建模为随机效应。

Notably, semantic representations are extracted from the same LALM under evaluation, enabling semantic control over variation as perceived by the model itself. 值得注意的是，语义表示是从正在评估的同一个 LALM 中提取的，从而能够对模型自身所感知的差异进行语义控制。

Experiments on simulated data and real-world benchmarks demonstrate that the proposed approach substantially reduces spurious fairness findings and yields more robust and interpretable estimates of subgroup performance differences. 在模拟数据和现实世界基准测试上的实验表明，该方法显著减少了虚假的公平性发现，并为子群体性能差异提供了更稳健且可解释的估计。