Measuring benchmark optimization in speech recognition

衡量语音识别中的基准测试优化

Public voice AI benchmarks increasingly suggest that models are performing at human levels. Yet those scores don’t always reflect how models work in the real-world. Since public benchmarks are open and widely used, models can also become optimized for the tests themselves. Their scores may improve because they have learned benchmark-specific patterns and not because they have become better at the underlying task. 公开的语音 AI 基准测试日益表明，模型表现已达到人类水平。然而，这些分数并不总是能反映模型在现实世界中的工作情况。由于公开基准测试是开放且被广泛使用的，模型可能会针对测试本身进行优化。它们的分数提高可能是因为学会了基准测试特定的模式，而不是因为在基础任务上变得更强。

One reason is that traditional benchmarks overlook many of the conditions and qualities that make voice systems reliable, natural, contextually appropriate, and effective in practice. That’s why we recently introduced held-out sets in Real World VoiceEQ, the Open-ASR Leaderboard, and the Far-field ASR Leaderboard: to measure more of what matters in real-world use. However, broader measurement alone does not solve the problem. 原因之一是传统基准测试忽略了许多使语音系统在实践中可靠、自然、语境恰当且有效的条件和质量。因此，我们最近在 Real World VoiceEQ、Open-ASR 排行榜和远场 ASR 排行榜中引入了留出集（held-out sets），旨在衡量更多现实使用中重要的指标。然而，仅靠更广泛的测量并不能解决问题。

This phenomenon, sometimes called benchmark optimization or “benchmaxxing,” is often discussed around machine learning, however, it has been difficult to measure in speech recognition. Our latest research introduces three tests to help quantify it. We evaluated 11 widely used open-source ASR models and found that several of the highest-scoring systems reproduced benchmark transcripts from the VoxPopuli English and LibriSpeech (clean, other) datasets – even when the audio contradicted them, relevant words had been silenced, or the audio equally supported two different written forms. 这种现象有时被称为基准测试优化或“刷榜”（benchmaxxing），在机器学习领域常被讨论，但在语音识别中却难以衡量。我们最新的研究引入了三项测试来帮助量化这一现象。我们评估了 11 个广泛使用的开源 ASR 模型，发现几个得分最高的系统会复现 VoxPopuli English 和 LibriSpeech（clean, other）数据集中的基准转录文本——即使音频内容与文本矛盾、相关词汇已被静音，或者音频同样支持两种不同的书写形式时也是如此。

In some cases, models appeared to rely not only on what was said, but also on subtle acoustic cues that indicated which benchmark they were being tested on. As a result, their scores overstated how well they could transcribe speech more generally. 在某些情况下，模型似乎不仅依赖于所说的内容，还依赖于微妙的声学线索，这些线索暗示了它们正在接受哪项基准测试。因此，它们的分数夸大了其在更广泛场景下转录语音的能力。

Reference disagreement (VoxPopuli case study)

参考答案分歧（VoxPopuli 案例研究）

VoxPopuli is known to contain a high number of transcription errors (which is why Artificial Analysis released a cleaned version). Our consensus disagreement probe tests what happens when leading ASR models encounter these errors: Do they accurately transcribe what the audio says, or reproduce the benchmark’s incorrect reference transcript? VoxPopuli 以包含大量转录错误而闻名（这也是 Artificial Analysis 发布清理版本的原因）。我们的共识分歧探测旨在测试领先的 ASR 模型在遇到这些错误时会发生什么：它们是准确转录音频内容，还是复现基准测试中错误的参考文本？

To test this at scale, we use an ensemble of independent models selected for their low phoneme error rate (PER). PER measures how closely a written transcription matches the sounds in the audio, making it a useful proxy for how faithfully a model transcribes what it hears. The ensemble results can be used to flag cases in which the models unanimously disagree with the benchmark’s reference transcript. We then compare a sample of those flagged cases against human annotations to validate the corrected transcripts. 为了大规模测试这一点，我们使用了一组因低音素错误率（PER）而被选中的独立模型集成。PER 衡量书面转录与音频声音的匹配程度，是衡量模型转录忠实度的有效指标。集成结果可用于标记模型一致不同意基准参考文本的情况。随后，我们将这些被标记案例的样本与人工标注进行对比，以验证修正后的转录文本。

For example, one VoxPopuli clip audibly includes the phrase “Thank you, Mr. President,” but the reference transcript omits “Thank you.” Six of the 11 models we tested reproduced the benchmark’s erroneous transcript—giving the “expected” answer even though it contradicted the audio. On the real clip, the formatting follows the same pattern: models that omit “Thank you” also reproduce the benchmark’s punctuation style, writing “Mr” without a period, while models that include the audible phrase tend to write “Mr.” with the period. 例如，一段 VoxPopuli 音频片段中清晰地包含了“Thank you, Mr. President”这一短语，但参考文本却漏掉了“Thank you”。在我们测试的 11 个模型中，有 6 个复现了基准测试的错误文本——即使这与音频矛盾，它们仍给出了“预期”答案。在真实片段中，格式遵循相同的模式：省略“Thank you”的模型也复现了基准测试的标点风格，即写成不带句点的“Mr”，而包含该短语的模型则倾向于写成带句点的“Mr.”。

When we present the same content in newly collected voices from EU parliamentary recordings or generic voices, this behavior often weakens or disappears. In the below samples, all but one model flips back to transcribing the audio-faithful transcript for a clone of a new parliamentary recording. This suggests that the models are responding to acoustic cues that help them identify the benchmark membership and thus produce the expected transcript even if it contradicts the audio. 当我们使用从欧盟议会录音中新采集的声音或通用声音呈现相同内容时，这种行为通常会减弱或消失。在下方的样本中，除一个模型外，其余模型在处理新的议会录音克隆时，都转回了忠实于音频的转录方式。这表明模型正在响应声学线索，这些线索帮助它们识别出基准测试的归属，从而产生预期的转录文本，即使这与音频内容相矛盾。