Trust but Verify: Designing Region-Inhomogeneous Forward Dynamics for Discrete Diffusion

Computer Science > Machine Learning arXiv:2610.22380 (cs) [Submitted on 2 Oct 2026] Title: Trust but Verify: Designing Region-Inhomogeneous Forward Dynamics for Discrete Diffusion Authors: Gunmay Jhingran.

计算机科学 > 机器学习 arXiv:2610.22380 (cs) [提交于 2026 年 10 月 2 日] 标题：信任但要验证：为离散扩散设计区域非均匀前向动力学 作者：Gunmay Jhingran。

Abstract: Continuous diffusion treats the forward process as a design space; discrete language diffusion often uses absorbing corruption, leaving context fully trusted throughout generation. This makes trust binary: corrupted context is copied as ground truth, propagating errors into generated spans.

摘要：连续扩散将前向过程视为一个设计空间；而离散语言扩散通常使用吸收式损坏，使得上下文在整个生成过程中被完全信任。这导致信任是二元的：损坏的上下文被复制为真实值，从而将错误传播到生成的片段中。

We reopen this design space by combining absorbing and uniform-mixing kernels with region-dependent rates. Missing positions follow absorbing dynamics, while context undergoes low-rate random substitution, exposing the model to observed but uncertain tokens during training.

我们通过结合吸收核和均匀混合核以及区域相关速率，重新开启了这个设计空间。缺失位置遵循吸收动力学，而上下文则经历低速率的随机替换，使模型在训练过程中接触到已观察到但具有不确定性的标记。

At inference, a heuristic sampler re-injects noise into suspicious context under an annealed intensity and freezes context updates late in the trajectory. An 8,000-step finetune of a pretrained 170M-parameter masked diffusion model takes under five hours on a 6 GB laptop GPU.

在推理时，启发式采样器会在退火强度下向可疑上下文重新注入噪声，并在轨迹后期冻结上下文更新。在一个 6 GB 笔记本电脑 GPU 上，对预训练的 1.7 亿参数掩码扩散模型进行 8,000 步微调耗时不到五小时。

The base model assigns roughly 50% probability to observed tokens even when they are uniform-random garbage. For uniform-random corruption, finetuning detects corrupted context with 0.96 recall and 0.91 precision, improves infill token-F1 by 3.8 points at 10% corruption and 6.4 points at 20%, and removes the accuracy dip next to corruptions.

即使在观察到的标记是均匀随机垃圾数据时，基础模型也会为其分配大约 50% 的概率。对于均匀随机损坏，微调后的模型能以 0.96 的召回率和 0.91 的精确率检测出损坏的上下文，在 10% 损坏率下将填充标记的 F1 值提高了 3.8 个点，在 20% 损坏率下提高了 6.4 个点，并消除了损坏处附近的准确率下降。

Clean conditioning shows no measurable regression, with a false-edit rate below 0.4%. Gains for plausible corruptions are weaker: about 0.5 F1 points at 10% corruption, without statistical significance.

干净的条件设置没有显示出可测量的回归，错误编辑率低于 0.4%。对于合理损坏的增益较弱：在 10% 损坏率下约为 0.5 个 F1 点，且不具有统计学意义。

Our results suggest that detectability and damage depend jointly on departure from the data conditional: nearby noise is hard to detect and semantically mild, while distant noise is conspicuous and destructive.

我们的结果表明，可检测性和损害程度共同取决于对数据条件分布的偏离程度：附近的噪声难以检测且语义影响较轻，而远处的噪声则显眼且具有破坏性。

We interpret absorbing diffusion as the infinite-contrast limit of a trust axis, analogous to the signal-to-noise gap between known and unknown regions in diffusion inpainting. In our setting of a 170M model and a single text domain, intermediate trust is reached by changing forward dynamics rather than architecture.

我们将吸收扩散解释为信任轴的无限对比度极限，这类似于扩散修复中已知区域和未知区域之间的信噪比差距。在我们 1.7 亿参数模型和单一文本域的设置中，通过改变前向动力学而非架构来实现中间信任。