Apple’s new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

苹果全新的 SpeechAnalyzer API：对比 Whisper 及其前代产品的基准测试

The result, up front: Apple’s new SpeechAnalyzer is the most accurate on-device speech engine we tested. It beat every Whisper model we ship, including Whisper Small, on both the clean and the noisy half of LibriSpeech, while running roughly three times faster than Small. And the API it replaces, SFSpeechRecognizer, came last on clean speech: behind even Whisper Tiny, a 40MB model. 首先公布结果：苹果全新的 SpeechAnalyzer 是我们测试过的最精准的端侧语音引擎。在 LibriSpeech 的纯净语音和噪声语音测试中，它击败了我们提供的所有 Whisper 模型（包括 Whisper Small），且运行速度比 Small 快约三倍。而它所取代的旧版 API——SFSpeechRecognizer，在纯净语音测试中表现垫底，甚至落后于仅 40MB 的 Whisper Tiny 模型。

Why we ran this

我们为何进行此项测试

With iOS 18 and macOS 15, Apple replaced SFSpeechRecognizer with a new API, SpeechAnalyzer and SpeechTranscriber. It published no accuracy figures for either one. So every developer deciding whether to migrate, and everyone comparing Apple’s built-in recognition against Whisper, has been guessing. We ship both Apple engines and three Whisper models side by side in Inscribe, a private on-device AI workspace, which puts us in an unusual position: we can run all five through identical production code paths on the same machine and the same audio. So we did. 随着 iOS 18 和 macOS 15 的发布，苹果用全新的 API（SpeechAnalyzer 和 SpeechTranscriber）取代了 SFSpeechRecognizer，但并未公布任何准确率数据。因此，无论是决定是否迁移的开发者，还是想要对比苹果内置识别功能与 Whisper 的用户，都只能靠猜。由于我们在端侧 AI 工作空间 Inscribe 中同时集成了苹果的两个引擎和三个 Whisper 模型，这使我们处于一个独特的地位：我们可以在同一台机器上，使用相同的音频，通过完全一致的生产代码路径运行这五个引擎。于是，我们进行了测试。

Should you migrate off SFSpeechRecognizer? Yes.

你应该弃用 SFSpeechRecognizer 吗？是的。

This is the clearest result in the data. The new API cuts word error rate by 3.5 to 4x on the same audio: from 9.02% to 2.12% on clean speech, and from 16.25% to 4.56% on noisy speech. There is no accuracy trade-off to weigh; the new API wins everywhere we measured, and it produces punctuated, cased text where the legacy engine’s output is rougher. Put differently: an hour-long meeting transcribed with the legacy API contains roughly four times as many wrong words as the same meeting through SpeechAnalyzer. If your app still uses SFSpeechRecognizer for anything longer than a voice command, the migration is worth it on accuracy alone. 这是数据中最明确的结论。在处理相同音频时，新 API 将词错误率（WER）降低了 3.5 到 4 倍：纯净语音从 9.02% 降至 2.12%，噪声语音从 16.25% 降至 4.56%。无需权衡准确率的取舍；新 API 在我们测试的所有维度上均胜出，且输出的文本带有标点和大小写，而旧引擎的输出则较为粗糙。换句话说：使用旧版 API 转录一小时的会议，其错误词数大约是使用 SpeechAnalyzer 转录时的四倍。如果你的应用仍在使用 SFSpeechRecognizer 处理语音指令以外的任务，仅凭准确率的提升，迁移就是值得的。

SpeechAnalyzer vs Whisper

SpeechAnalyzer 对比 Whisper

The more surprising result: Apple’s new engine also beat Whisper Small, the largest model we ship, by a comfortable margin on both splits, at roughly a third of Whisper Small’s compute time per second of audio. For English, on Apple hardware, the built-in engine is now the strongest on-device option we can measure. Whisper keeps two real advantages. It covers far more languages (SpeechTranscriber supports around 30 locales), and it runs anywhere, not just on Apple platforms with OS 18+. But for English transcription on a current iPhone or Mac, the days of Whisper being the automatic accuracy pick are over. We changed our own product on this result: Inscribe’s Auto engine now prefers SpeechAnalyzer for the languages it supports, and Whisper for everything else. 更令人惊讶的结果是：苹果的新引擎在两个测试维度上都以明显的优势击败了我们提供的最大模型 Whisper Small，且每秒音频的计算耗时仅为 Whisper Small 的三分之一左右。在苹果硬件上处理英语时，内置引擎已成为我们所能测得的最强端侧方案。Whisper 仍保留了两个真正的优势：它支持的语言种类远多于苹果（SpeechTranscriber 仅支持约 30 种语言环境），且它可以在任何地方运行，而不局限于搭载 iOS 18+ 的苹果平台。但对于当前 iPhone 或 Mac 上的英语转录而言，Whisper 作为“准确率首选”的时代已经结束。基于此结果，我们调整了自己的产品：Inscribe 的自动引擎现在会优先选择 SpeechAnalyzer（针对其支持的语言），其余语言则使用 Whisper。

Methodology, and why you can check it

方法论及可验证性

A benchmark from a company that sells one of the engines should be treated with suspicion. Ours has two properties designed for that suspicion. The Whisper column is reproducible against OpenAI’s own numbers. We used LibriSpeech precisely because OpenAI published Whisper’s WER on it. If our harness measured Whisper correctly, our numbers should land on theirs. They do, on all six measurements. The small, consistent positive offset (a slightly stricter text normalizer plus CoreML quantization) is what honest reproduction looks like; random error would scatter in both directions. Since the same corpus, normalizer, and scorer produced the Apple columns, the numbers nobody else can check inherit the validation from the numbers anyone can. 由销售其中一款引擎的公司所做的基准测试应当受到质疑。我们的测试具备两个旨在应对这种质疑的特性。Whisper 的数据列可以与 OpenAI 官方数据进行复现对比。我们使用 LibriSpeech 正是因为 OpenAI 在该数据集上发布了 Whisper 的 WER 数据。如果我们的测试框架测量准确，我们的数据应该与他们的吻合。事实确实如此，六项测量结果均吻合。微小且一致的正向偏差（源于更严格的文本归一化处理及 CoreML 量化）正是诚实复现的体现；如果是随机误差，偏差会向两个方向散布。由于苹果引擎的数据列采用了相同的语料库、归一化器和评分器，那些无法被他人直接验证的数据，也通过可验证的数据获得了背书。