Token Time Continuous Diffusion for Language Modeling

语言建模的 Token 时间连续扩散模型

In this paper we introduce token time continuous diffusion (TTCD), a new diffusion language model which (a) operates in continuous space, deterministically mapping Gaussian noise to a final token canvas with no further sampling, and crucially (b) incorporates a new notion of per-token times, with some tokens proceeding from noise to token at a faster rate than others.

在本文中，我们引入了 Token 时间连续扩散模型（TTCD），这是一种新型扩散语言模型。它具有以下特点：(a) 在连续空间中运行，将高斯噪声确定性地映射到最终的 Token 画布上，无需进一步采样；(b) 关键在于引入了“每个 Token 的时间”这一新概念，使得某些 Token 从噪声到实体的演化速率比其他 Token 更快。

Continuous space modeling helps TTCD avoid the parallel sampling of multiple tokens, which is a key source of inaccuracy at high speedups for models that iterate purely in discrete space.

连续空间建模帮助 TTCD 避免了对多个 Token 进行并行采样，而并行采样正是那些纯粹在离散空间中迭代的模型在高加速比下产生不准确性的主要来源。

The notion of per-token times helps TTCD to better model conditional generation, allows for more sure tokens to proceed at a faster rate, and allows for differentiated inter-token influences during refinement.

“每个 Token 的时间”这一概念有助于 TTCD 更好地建模条件生成，允许确定性更高的 Token 以更快的速度演化，并在精炼过程中实现 Token 之间差异化的相互影响。

TTCD outperforms discrete models at high speedups. We train a 160M parameter TTCD model on OpenWebText, and then self-distill it; we find that at high speedups we are comparable in unconditional generation quality, and outperform in conditional generation, several existing models of similar size trained on the same data, and self-distilled. We achieve similar gains in Sudoku solving as well.

在高加速比下，TTCD 的表现优于离散模型。我们在 OpenWebText 上训练了一个 1.6 亿参数的 TTCD 模型，并对其进行了自蒸馏；研究发现，在高加速比下，我们的无条件生成质量与同等规模、在相同数据上训练并自蒸馏的现有模型相当，而在条件生成方面则表现更优。我们在数独求解任务中也取得了类似的性能提升。