Causal Analysis and Mitigation of Spurious Onsets in Full-Duplex Speech LLMs
Causal Analysis and Mitigation of Spurious Onsets in Full-Duplex Speech LLMs
全双工语音大模型中虚假语音起点的因果分析与缓解
Abstract: Speech-to-speech LLMs like Moshi, and its derivative PersonaPlex, can listen and speak concurrently through full-duplex generation. However, they can begin speaking inappropriately during prolonged user silence: under digital-zero input, Moshi and PersonaPlex initiate speech in 12/40 and 11/40 five-minute continuations, respectively.
摘要： 像 Moshi 及其衍生模型 PersonaPlex 这样的语音到语音（Speech-to-speech）大模型，可以通过全双工生成实现同时听取和说话。然而，它们在用户长时间沉默时可能会不适当地开始说话：在数字零输入（digital-zero input）的情况下，Moshi 和 PersonaPlex 在 40 次五分钟的连续测试中，分别有 12 次和 11 次触发了虚假语音。
What causes this spurious speech? We investigate two hypotheses: either repeated sampling selects speech despite persistently low onset probabilities, or conditioning on the model’s nonspeech outputs causes an abrupt spike in onset probability. We find that, at every observed onset, speech probability spikes by over nine orders of magnitude in one 80-ms frame, supporting the latter hypothesis.
这种虚假语音的原因是什么？我们研究了两个假设：一是尽管语音起点概率持续较低，但重复采样仍选择了语音；二是模型对自身非语音输出的条件化导致了起点概率的突然激增。我们发现，在每一个观察到的起点处，语音概率在 80 毫秒的帧内激增了超过九个数量级，这支持了后一种假设。
Then, to suppress these onsets without blocking genuine responses, we ask a causal counterfactual question: is the model responding to user speech, or would its next-token distribution remain similar if the preceding user input were muted? Accordingly, we suppress onsets whose distributions change little under this intervention.
随后，为了在不阻碍真实响应的前提下抑制这些虚假起点，我们提出了一个因果反事实问题：模型是在响应用户语音，还是说如果将之前的用户输入静音，其下一个 token 的分布是否会保持不变？据此，我们抑制了在进行这种干预后分布变化不大的起点。
Across 40 held-out trials per model with realistic microphone noise, our method suppresses 13/13 Moshi and 9/9 PersonaPlex spurious onsets, while preserving 40/40 genuine responses per model. Our inference-time method requires no retraining and runs in real-time, with 95th-percentile decision time below 61 ms, within the 80-ms frame budget. Our code is available at this https URL.
在包含真实麦克风噪声的每个模型 40 次留出试验中，我们的方法成功抑制了 Moshi 的 13/13 次和 PersonaPlex 的 9/9 次虚假起点，同时保留了每个模型 40/40 次的真实响应。我们的推理阶段方法无需重新训练，且能实时运行，95 分位决策时间低于 61 毫秒，处于 80 毫秒的帧预算之内。我们的代码可在该链接获取。