Revisiting the Relation Between Language Model Perplexity and ASR Word Error Rate for Modern End-to-End Speech Recognition

重新审视现代端到端语音识别中语言模型困惑度与 ASR 词错误率之间的关系

Abstract: Language model (LM) perplexity (PPL) has historically been used as a proxy for automatic speech recognition (ASR) word error rate (WER), with prior work reporting an approximately linear relation in log-log space. 摘要： 语言模型（LM）的困惑度（PPL）在历史上一直被用作自动语音识别（ASR）词错误率（WER）的代理指标，既往研究表明两者在对数-对数空间中呈近似线性关系。

Modern end-to-end ASR systems challenge this assumption because they already contain internal language modeling capacity, are often evaluated without external language models, and can now be combined with neural LMs and large language models (LLMs) through different recognition strategies. 现代端到端 ASR 系统对这一假设提出了挑战，因为它们本身已经具备了内部语言建模能力，通常在没有外部语言模型的情况下进行评估，并且现在可以通过不同的识别策略与神经语言模型及大语言模型（LLM）相结合。

This paper revisits the relation between PPL and WER for modern ASR systems. We study whether external LMs still improve current end-to-end ASR systems, whether the PPL-WER relation remains linear in log-log space, how encoder context length affects this relation, and how LLM perplexities fit into the trend observed for standard neural LMs. 本文重新审视了现代 ASR 系统中 PPL 与 WER 之间的关系。我们研究了外部语言模型是否仍能提升当前的端到端 ASR 系统，PPL-WER 关系在对数-对数空间中是否依然保持线性，编码器上下文长度如何影响这种关系，以及 LLM 的困惑度如何符合标准神经语言模型所观察到的趋势。

We further investigate internal language modeling (ILM) in attention-based encoder-decoder systems and show that ILM subtraction changes the observed PPL-WER relation, indicating that the decoder’s internal LM must be considered when interpreting the effect of external LM quality. 我们进一步研究了基于注意力的编码器-解码器系统中的内部语言建模（ILM），并表明 ILM 减法会改变所观察到的 PPL-WER 关系，这说明在解释外部语言模型质量的影响时，必须考虑解码器的内部语言模型。