Token Merging for Multilingual Speech Recognition: A Systematic Study Across Model Scale and Fine-Tuning

用于多语言语音识别的 Token 合并技术：跨模型规模与微调的系统性研究

Abstract: Leading multilingual speech recognition models like Whisper transcribe diverse, low-resource languages without language-specific training but are computationally expensive to deploy. Token merging mitigates this inefficiency by dynamically combining redundant features, shortening the sequence length during inference without requiring retraining.

摘要： 像 Whisper 这样的领先多语言语音识别模型，无需针对特定语言进行训练即可转录各种低资源语言，但其部署的计算成本高昂。Token 合并（Token Merging）通过动态合并冗余特征来缓解这种低效问题，在无需重新训练的情况下缩短了推理过程中的序列长度。

In this paper, we systematically evaluate token merging on the Whisper model family across sixteen diverse languages and three different model sizes. We also test how token merging interacts with fine-tuning (DoRA) on low-resource languages.

在本文中，我们针对 Whisper 模型系列，在十六种不同的语言和三种不同的模型规模上对 Token 合并技术进行了系统性评估。我们还测试了 Token 合并与低资源语言微调（DoRA）之间的相互作用。

Our findings show that merging tokens increases computational efficiency with almost no loss in transcription accuracy across most low-resource languages and model sizes, and it works even after the model has been fine-tuned. Our results demonstrate that token merging is a highly practical method for making multilingual speech recognition faster and cheaper to deploy.

研究结果表明，在大多数低资源语言和模型规模下，合并 Token 可以在几乎不损失转录准确率的前提下提高计算效率，且该方法在模型经过微调后依然有效。我们的研究结果证明，Token 合并是一种使多语言语音识别部署更快、成本更低的实用方法。