Introducing Real World VoiceEQ: Measuring the human quality of voice AI
Introducing Real World VoiceEQ: Measuring the human quality of voice AI
介绍 Real World VoiceEQ：衡量语音 AI 的人类品质
Existing benchmarks suggest voice AI is nearing human-level performance but real-world conversations tell a different story. Voice is rapidly becoming AI’s primary interface. From customer support and healthcare to education, entertainment, and personal assistants, speech is increasingly replacing text as the way people interact with AI. 现有的基准测试表明，语音 AI 正接近人类水平，但现实世界的对话却呈现出不同的情况。语音正迅速成为 AI 的主要交互界面。从客户支持和医疗保健到教育、娱乐和个人助理，语音正日益取代文本，成为人们与 AI 交互的主要方式。
Over the last few years, voice models have improved dramatically. Word error rates continue to fall, latency has reached conversational speeds, and many established benchmarks are approaching saturation. Yet anyone who regularly uses voice AI knows something still feels off. Voice models can sound like different people over the course of a conversation, miss hesitation or uncertainty, and struggle with accents, noise, or emotional speech. Those shortcomings are easy to miss in benchmarks focused on latency and word error rate. People care whether a voice system can truly listen, respond appropriately, and remain natural and reliable in real conversations. 在过去几年中，语音模型取得了巨大的进步。词错率持续下降，延迟已达到对话速度，许多既定的基准测试也已接近饱和。然而，任何经常使用语音 AI 的人都知道，总感觉有些不对劲。语音模型在对话过程中听起来可能像不同的人，无法捕捉犹豫或不确定性，并且在处理口音、噪音或情感语音时表现吃力。这些缺陷在侧重于延迟和词错率的基准测试中很容易被忽略。人们关心的是语音系统是否能真正倾听、做出适当回应，并在真实对话中保持自然和可靠。
A broader benchmark for voice AI
更广泛的语音 AI 基准测试
To measure those qualities, we built Real World VoiceEQ—a benchmark designed to evaluate the human quality of voice interaction. It assesses whether voice systems can recognize, produce, and respond to the acoustic information transcripts leave out, from tone and emotion to speaker identity and background context. Real World VoiceEQ evaluates more than 40 leading proprietary and open-source voice models across 15+ key evaluation dimensions and more than 60 metrics spanning Automatic Speech Recognition (ASR), Text-to-Speech (TTS), Speech-to-Speech (S2S), and Speech Understanding. 为了衡量这些品质，我们构建了 Real World VoiceEQ——一个旨在评估语音交互人类品质的基准测试。它评估语音系统是否能够识别、生成并响应转录文本所遗漏的声学信息，从语调和情感，到说话人身份和背景语境。Real World VoiceEQ 评估了 40 多种领先的专有和开源语音模型，涵盖了 15 个以上的关键评估维度和 60 多个指标，跨越了自动语音识别 (ASR)、文本转语音 (TTS)、语音转语音 (S2S) 和语音理解等领域。
Real World VoiceEQ was developed from more than 1 million individual human ratings collected across different demographics, speaking styles, and acoustic environments. The current benchmark includes 785,000 TTS ratings and 48,000 STS ratings, making it one of the largest human evaluations of voice AI conducted to date. Every evaluation was conducted using Kairos, our flexible, voice-native evaluation platform. The same infrastructure enables frontier AI labs and enterprises to run custom evaluations tailored to specific use cases, identify granular failure modes in production voice systems, generate human preference data, and continuously improve models through reinforcement learning and human feedback. Real World VoiceEQ 是基于从不同人口统计学特征、说话风格和声学环境中收集的超过 100 万次个人人类评分开发而成的。目前的基准测试包含 785,000 次 TTS 评分和 48,000 次 STS 评分，使其成为迄今为止规模最大的人类语音 AI 评估之一。每次评估均使用我们灵活的语音原生评估平台 Kairos 进行。同一基础设施使前沿 AI 实验室和企业能够针对特定用例运行自定义评估，识别生产语音系统中的细粒度故障模式，生成人类偏好数据，并通过强化学习和人类反馈持续改进模型。
Key findings from Real World VoiceEQ
Real World VoiceEQ 的主要发现
Progress in voice AI is becoming increasingly specialized. The race for a single “best” voice model is giving way to a collection of specialized capabilities. Today’s leading systems optimize for different strengths—including technical accuracy, emotional understanding, conversational intelligence, expressiveness, and robustness. One model that excels at repeating booking reference numbers, bank account details, or complex pharmaceutical names may struggle to produce emotionally expressive speech. Another may sound remarkably natural but be less reliable on precision-oriented tasks. As voice AI matures, measuring progress increasingly requires evaluating these capabilities independently rather than collapsing them into a single overall score. In our TTS evaluations, no system configuration ranked among the top five across all eight capability groups—underscoring why there is no single “best” voice model. 语音 AI 的进步正变得日益专业化。对单一“最佳”语音模型的追逐，正让位于一系列专业能力的集合。当今领先的系统针对不同的优势进行优化，包括技术准确性、情感理解、对话智能、表现力和鲁棒性。一个擅长重复预订参考号、银行账户详情或复杂药品名称的模型，可能在生成富有情感的语音时表现吃力。另一个模型听起来可能非常自然，但在注重精确性的任务上可靠性较低。随着语音 AI 的成熟，衡量进步越来越需要独立评估这些能力，而不是将它们合并为一个单一的总分。在我们的 TTS 评估中，没有任何系统配置在所有八个能力组中都排名前五，这强调了为什么不存在单一的“最佳”语音模型。
Voice models have become better at speaking than actually listening. Speech-to-Speech models showed the widest variation of any category we evaluated. Some systems recognized emotion exceptionally well but struggled to respond naturally. We found that access to audio did not guarantee that agents used the paralinguistic information it contained. Some systems remained largely transcript-driven, relying on the words being spoken while overlooking cues such as tone, pacing, hesitation, emphasis, and volume. Humans naturally use these cues to infer confidence, uncertainty, frustration, sarcasm, and empathy. Today’s models often miss them. Imagine a banking agent asking whether you recognize a potentially fraudulent transaction. A confident “Yes” and a hesitant “…yes…” may have completely different meanings, even though the transcript is identical. Humans recognize that difference immediately. Many of today’s voice models do not. 语音模型在说话方面已经比实际倾听做得更好。语音转语音 (S2S) 模型在我们评估的所有类别中表现出最大的差异。一些系统能非常好地识别情感，但在自然回应方面却很吃力。我们发现，能够获取音频并不保证智能体能利用其中包含的副语言信息。一些系统在很大程度上仍然是基于转录文本驱动的，依赖于所说的词汇，却忽略了语调、语速、犹豫、强调和音量等线索。人类自然地利用这些线索来推断自信、不确定性、挫败感、讽刺和同理心。当今的模型往往会错过这些。想象一下，银行客服询问你是否识别一笔潜在的欺诈交易。自信的“是的”和犹豫的“……是的……”可能有着完全不同的含义，尽管转录文本完全相同。人类能立即识别出这种差异，而当今的许多语音模型却做不到。
Traditional benchmarks increasingly overestimate real-world performance. Many established benchmarks are nearing their limits and don’t reflect real-world conditions. Models still struggle with accented speech, overlapping speakers, emotion, background noise, and longer conversations. In our evaluation, performance varies far more across leading open-source and proprietary models than traditional benchmarks suggest. In one example, transcription word error rates on noise-backed speech were roughly four times higher than on music-backed speech, showing how a single background-audio score can hide the real failure mode. 传统的基准测试越来越高估了现实世界的表现。许多既定的基准测试已接近极限，无法反映现实世界的状况。模型在处理带口音的语音、重叠说话人、情感、背景噪音和较长对话时仍然表现吃力。在我们的评估中，领先的开源和专有模型之间的性能差异远比传统基准测试所显示的要大。举个例子，在有噪音背景的语音中，转录词错率大约是音乐背景语音的四倍，这表明单一的背景音频评分是如何掩盖真实故障模式的。
Human evaluation remains essential. In preliminary research, we found signs that some models may be optimized for established public benchmarks. Several reproduced known errors in reference transcripts, followed arbitrary spelling conventions, and even reconstructed masked words that were not present in the audio. LLMs are now widely used to evaluate text-based models, but our findings suggest that speech-language models (SLMs) should be used more carefully for voice evaluation. 人类评估仍然至关重要。在初步研究中，我们发现一些模型可能针对既定的公共基准测试进行了优化。一些模型重现了参考转录中的已知错误，遵循了随意的拼写惯例，甚至重构了音频中不存在的被掩盖的词汇。大语言模型 (LLM) 现在被广泛用于评估基于文本的模型，但我们的研究结果表明，在进行语音评估时，应更谨慎地使用语音语言模型 (SLM)。