Scalable and Culturally Specific Stereotype Dataset Construction via Human-LLM Collaboration

通过人机协作构建可扩展且具有文化特异性的刻板印象数据集

Abstract: Research on stereotypes in large language models (LLMs) has largely focused on English-speaking contexts, due to the lack of datasets in other languages and the high cost of manual annotation in underrepresented cultures. 摘要： 由于缺乏其他语言的数据集以及在代表性不足的文化中进行人工标注的高昂成本，目前关于大语言模型（LLM）中刻板印象的研究主要集中在英语语境下。

To address this gap, we introduce a cost-efficient human-LLM collaborative annotation framework and apply it to construct EspanStereo, a Spanish-language stereotype dataset spanning multiple Spanish-speaking countries across Europe and Latin America. 为了弥补这一差距，我们引入了一种经济高效的人机协作标注框架，并将其应用于构建 EspanStereo——一个涵盖欧洲和拉丁美洲多个西班牙语国家的西班牙语刻板印象数据集。

EspanStereo captures both well-documented stereotypes from prior literature and culturally specific biases absent from English-centric resources. EspanStereo 不仅捕捉了先前文献中记录详尽的刻板印象，还涵盖了以英语为中心的资源中所缺失的、具有文化特异性的偏见。

Using LLMs to generate candidate stereotypes and in-culture annotators to validate them, we demonstrate the framework’s effectiveness in identifying nuanced, region-specific biases. 通过利用大语言模型生成候选刻板印象，并由本土文化背景的标注者进行验证，我们证明了该框架在识别细微的、特定区域偏见方面的有效性。

Our evaluation of Spanish-supporting LLMs using EspanStereo reveals significant variation in stereotypical behavior across countries, highlighting the need for more culturally grounded assessments. 我们使用 EspanStereo 对支持西班牙语的大语言模型进行了评估，结果显示不同国家之间的刻板印象行为存在显著差异，这凸显了进行更具文化基础的评估的必要性。

Beyond Spanish, our framework is adaptable to other languages and regions, offering a scalable path toward multilingual stereotype benchmarks. 除了西班牙语之外，我们的框架还适用于其他语言和地区，为实现多语言刻板印象基准测试提供了一条可扩展的路径。

This work broadens the scope of stereotype analysis in LLMs and lays the groundwork for comprehensive cross-cultural bias evaluation. 这项工作拓宽了大语言模型中刻板印象分析的范围，并为全面的跨文化偏见评估奠定了基础。