Gradient-Based Speech-to-Text Alignment for Any ASR Model: From CTC to Speech LLMs

基于梯度的语音转文本对齐：适用于任何 ASR 模型（从 CTC 到语音大模型）

Abstract: Speech-to-text alignment means finding the temporal boundaries of each word in the audio. Some models provide such an alignment directly and others do not. Connectionist temporal classification (CTC) and transducer models have an alignment by construction, whereas attention-based encoder-decoders (AED) and speech large language models (LLMs) do not, and their word timings are usually read off the attention weights instead.

摘要： 语音转文本对齐是指确定音频中每个单词的时间边界。有些模型可以直接提供这种对齐，而另一些则不能。联结主义时间分类（CTC）和转换器（Transducer）模型在构建时就具备对齐能力，而基于注意力的编码器-解码器（AED）和语音大语言模型（LLM）则不具备，它们的单词时间戳通常是通过读取注意力权重来获取的。

All of these signals live on the encoder frame grid, which bounds their temporal precision. We study a generic gradient-based alignment that applies to any differentiable ASR model. We take the gradient of each teacher-forced token log probability with respect to the input, reduce it to a per-frame saliency, and decode the resulting matrix into word boundaries with a single dynamic-programming pass.

所有这些信号都存在于编码器的帧网格上，这限制了它们的时间精度。我们研究了一种通用的、基于梯度的对齐方法，该方法适用于任何可微分的 ASR 模型。我们计算每个教师强制（teacher-forced）标记对数概率相对于输入的梯度，将其简化为逐帧显著性（saliency），并通过单次动态规划过程将生成的矩阵解码为单词边界。

The method needs no training, no model modification and no alignment heads, works across all model families including the speech LLMs, and aligns on the input grid rather than on the coarser encoder grid. We evaluate it on sixteen models from four families, on read (TIMIT) and spontaneous (Buckeye) speech, each against the model’s own native or attention-based alignment.

该方法无需训练、无需修改模型、也无需对齐头（alignment heads），适用于包括语音大模型在内的所有模型系列，并且是在输入网格而非较粗糙的编码器网格上进行对齐。我们在来自四个系列的十六个模型上进行了评估，涵盖朗读语音（TIMIT）和自发语音（Buckeye），并将结果与模型自身原生的或基于注意力的对齐方式进行了对比。

We find that the gradient yields a usable alignment for every model, that it is usually somewhat behind a strong native aligner but better where the native alignment is weak, as for the streaming models, and that its main disadvantage is the cost of one backward pass per token.

我们发现，该梯度方法为每个模型都提供了可用的对齐结果；虽然它通常略逊于强大的原生对齐器，但在原生对齐较弱的情况下（如流式模型）表现更好；其主要缺点是每个标记需要进行一次反向传播计算。