The Open ASR Leaderboard Adds Its First Global South Language
The Open ASR Leaderboard Adds Its First Global South Language
Open ASR 排行榜迎来首个“全球南方”语言
Voice Arena and Hugging Face partner to launch open ASR evaluation for Hindi and Indian English. Voice Arena 与 Hugging Face 合作，为印地语和印度英语推出开放式自动语音识别（ASR）评估。
Benchmarks decide what gets built. A model that scores well on the Open ASR Leaderboard gets adopted and iterated on, while capabilities the leaderboard does not measure tend not to improve. Much of the recent work on the leaderboard has gone into making the evaluation metrics more trustworthy: Held-out private splits. Benchmark-fitting analysis to quantify how much models are reproducing reference transcripts rather than transcribing solely on the audio. Closing the gaps in normalisers to ensure correct predictions/variants are not penalized. All of that makes one number (WER) harder to game. It is still one number. 基准测试决定了技术的发展方向。在 Open ASR 排行榜上得分高的模型会被采用并迭代，而排行榜未衡量的能力往往得不到提升。近期排行榜的大部分工作都致力于使评估指标更加可信：包括保留私有数据集、通过基准拟合分析来量化模型是在复现参考文本还是在真正转录音频，以及完善归一化工具以确保正确的预测或变体不会被扣分。所有这些努力使得单一指标（WER，词错误率）更难被“刷分”，但它终究仍是一个单一指标。
A long line of work has shown that ASR error rates are not evenly distributed across the people using them. Racial disparities in automated speech recognition found commercial systems roughly twice as bad for Black speakers as for white speakers, and Quantifying Bias in Automatic Speech Recognition found further differences by gender, age and accent. None of that is visible on a leaderboard, and not because the leaderboard is hiding it. The test sets it runs on record what was said and almost nothing about who said it. 大量研究表明，ASR 的错误率在不同用户群体间的分布并不均匀。针对自动语音识别中种族差异的研究发现，商业系统对黑人说话者的识别效果大约比白人差两倍；而《量化自动语音识别中的偏见》一文则进一步发现了性别、年龄和口音带来的差异。这些在排行榜上都无法体现，并非因为排行榜刻意隐瞒，而是因为其测试集只记录了“说了什么”，几乎没有记录“是谁说的”。
To address this gap, we introduce two evaluation sets to the Open ASR Leaderboard: Monsoon en-IN and Monsoon hi-IN. Hindi, spoken by more than half a billion people, is the first Indic language on a multilingual tab that currently covers only European languages. Each set is released as a public split, available for self-scoring, and a private split withheld to limit benchmark-specific optimisation. The four splits are speaker-disjoint, comprising 4,888 speakers, with 12 speaker attributes recorded for each. 为了弥补这一差距，我们在 Open ASR 排行榜中引入了两个评估集：Monsoon en-IN 和 Monsoon hi-IN。印地语拥有超过五亿的使用者，是该多语言标签页中首个印度语言（此前仅涵盖欧洲语言）。每个数据集都分为公开集（可供自我评分）和私有集（不对外公开，以限制针对基准测试的优化）。这四个数据集在说话者层面互不重叠，共包含 4,888 名说话者，并为每位说话者记录了 12 项属性。
Design of the collection
数据采集设计
A test set can only expose a failure mode it varies along. Most benchmarks are built from whatever audio was readily available. Monsoon was built to vary along nine axes: geography, age, gender, vocabulary, devices, acoustic environments, speech type, speech rate, and the existence of multiple valid transcripts for the same audio. Each is a way an aggregate WER can be right on average and wrong for a particular population. The collection method follows from that. 测试集只能暴露其覆盖范围内的故障模式。大多数基准测试是基于现成的音频构建的，而 Monsoon 的设计旨在涵盖九个维度：地理位置、年龄、性别、词汇、设备、声学环境、语音类型、语速，以及同一音频是否存在多个有效转录文本。每一个维度都可能导致总体的 WER 在平均水平上表现良好，但在特定群体中却表现糟糕。我们的采集方法正是基于此设计的。
Geography comes from recruiting across hundreds of districts rather than recording longer sessions in fewer places. Devices and acoustic conditions come from contributors using their own handsets and connections, indoors and out, rather than supplied hardware in a quiet room. Vocabulary, speech type and speech rate come from the prompts: everyday topics that push contributors toward opinion, disagreement, narration and recall, which is where named entities, numbers and unrehearsed phrasing appear. Age and gender are recorded per speaker and verified. Multiple valid transcripts is a property of the reference rather than the audio, and it is the subject of a later section. 地理多样性通过在数百个地区招募人员实现，而非在少数几个地点进行长时间录音。设备和声学条件则源于贡献者使用自己的手机和网络，在室内外不同环境下录制，而非在安静房间内使用统一硬件。词汇、语音类型和语速则通过提示词引导：日常话题促使贡献者表达观点、进行辩论、叙述和回忆，这些场景中会出现命名实体、数字和未经排练的措辞。年龄和性别由每位说话者提供并经过核实。关于“多个有效转录文本”，这是参考文本的属性而非音频本身的属性，我们将在后续章节讨论。
Dataset composition
数据集构成
The data is sourced from unscripted dual-channel spontaneous conversations, with clips segmented from a single channel so that each clip carries one speaker. Along with the fields reported in the table, each clip also records occupation, education, marital status, income band, handset brand, current city and years in the current district. 数据源自无脚本的双通道自发对话，通过从单通道分割音频片段，确保每个片段仅包含一名说话者。除了表格中列出的字段外，每个片段还记录了职业、教育程度、婚姻状况、收入区间、手机品牌、当前城市以及在当前地区居住的年限。
The English sets use standard string references, where the leaderboard’s normaliser collapses most spelling variation. Hindi has far more of it, and no normaliser can resolve it, because the variants are not a fixed mapping between two conventions. The Hindi sets therefore ship a lattice: for each span of the transcript, a list of the spellings that are accepted as correct. 英语数据集使用标准的字符串参考，排行榜的归一化工具可以处理大部分拼写变体。印地语的变体要多得多，且没有任何归一化工具能完全解决，因为这些变体并非两种规范之间的固定映射。因此，印地语数据集提供了一个“晶格”（lattice）：对于转录文本的每一段，都列出了所有被认可的正确拼写。
Speaker coverage
说话者覆盖范围
Monsoon is small measured in hours and large measured in speakers. That is the design, and it is where most of the value sits. 以时长衡量，Monsoon 数据集规模较小；但以说话者人数衡量，它规模庞大。这正是我们的设计初衷，也是该数据集的核心价值所在。