The Generative Gap Between Speculative Drafter and Continuous Diffusion Language Model

Computer Science > Computation and Language arXiv:2610.40847 (cs) [Submitted on 29 Oct 2026] Title: The Generative Gap Between Speculative Drafter and Continuous Diffusion Language Model Authors: Yujian Chen, Hao Jiang, Guang Cheng, Weijian Luo, Tianyang Hu.

计算机科学 > 计算与语言 arXiv:2610.40847 (cs) [提交于 2026 年 10 月 29 日] 标题：推测草稿模型与连续扩散语言模型之间的生成差距 作者：Yujian Chen, Hao Jiang, Guang Cheng, Weijian Luo, Tianyang Hu。

Abstract: Continuous diffusion language models (CDLMs) and speculative drafters offer efficient language generation but are typically evaluated differently: CDLMs generate independently through iterative refinement, whereas drafters propose text for target-model verification and correction.

摘要：连续扩散语言模型（CDLM）和推测草稿模型提供了高效的语言生成方式，但它们的评估方式通常不同：CDLM 通过迭代细化独立生成文本，而草稿模型则为目标模型的验证和校正提供文本建议。

We compare their architectures without verification, using LangFlow for CDLMs and adapting DSpark’s Markov and recurrent-head architectures for standalone generation. All models are trained from scratch for unconditional, single-block OpenWebText generation at lengths 16, 128, and 1024, without target features or verification, using common preprocessing, backbone dimensions, and training-update budgets.

我们在不进行验证的情况下比较了它们的架构，使用 LangFlow 作为 CDLM，并调整了 DSpark 的马尔可夫和循环头架构以进行独立生成。所有模型均从零开始训练，用于长度为 16、128 和 1024 的无条件单块 OpenWebText 生成，不包含目标特征或验证，并使用通用的预处理、主干维度和训练更新预算。

We evaluate generation perplexity (GenPPL) with an external language model and distributional similarity to reference text with MAUVE. With default denoising steps equal to sequence length, LangFlow outperforms both DSpark variants on both metrics at lengths 128 and 1024, while DSpark-RNN leads at 16.

我们使用外部语言模型评估生成困惑度（GenPPL），并使用 MAUVE 评估与参考文本的分布相似度。在默认去噪步数等于序列长度的情况下，LangFlow 在长度为 128 和 1024 时在这两项指标上均优于两种 DSpark 变体，而 DSpark-RNN 在长度为 16 时表现领先。

A conditional KL analysis characterizes dependencies excluded by the Markov head’s previous-token conditioning. Recurrent memory removes this restriction and improves both metrics, but the longer-sequence gap persists. Teacher probability matching further lowers Markov GenPPL without observed MAUVE gains.

条件 KL 分析刻画了马尔可夫头的前一个标记条件所排除的依赖关系。循环记忆消除了这一限制并改善了这两项指标，但长序列差距依然存在。教师概率匹配进一步降低了马尔可夫模型的 GenPPL，但未观察到 MAUVE 的提升。

Increasing LangFlow’s denoising steps at fixed weights improves both metrics at lengths 16 and 128, changing the length-128 ordering relative to DSpark-RNN. These results establish LangFlow’s longer-sequence advantage under the evaluated budgets and highlight iterative full-sequence refinement as a route to improving standalone generation beyond gains from recurrent head memory.

在权重固定的情况下增加 LangFlow 的去噪步数，改善了长度为 16 和 128 时的两项指标，改变了长度为 128 时相对于 DSpark-RNN 的排名。这些结果确立了 LangFlow 在所评估预算下的长序列优势，并强调了迭代全序列细化是超越循环头记忆增益、提升独立生成能力的一种途径。