TraceQuery: learning what to recompute from full-denoising trajectory in diffusion language models

查看原文 / View Original 编辑 / Edit

Computer Science > Computation and Language arXiv:2610.33008 (cs) [Submitted on 7 Oct 2026] Title: TraceQuery: learning what to recompute from full-denoising trajectory in diffusion language models Authors: Yukai Jin, Zhankai Ye, Jiayue Yang, Fangyi Li, Shuoqiu Li, Xu Gao, Shibo Li, Xin Liu, Shangqian Gao.

计算机科学 > 计算与语言 arXiv:2610.33008 (cs) [提交于 2026 年 10 月 7 日] 标题：TraceQuery：从扩散语言模型的全去噪轨迹中学习重计算内容作者：Yukai Jin, Zhankai Ye, Jiayue Yang, Fangyi Li, Shuoqiu Li, Xu Gao, Shibo Li, Xin Liu, Shangqian Gao。

Abstract: Diffusion large language models (dLLMs) enable flexible generation through iterative masked denoising, but incur substantial inference cost because token representations are repeatedly recomputed across denoising steps. Existing cache-based methods reduce this cost by reusing intermediate results across denoising steps and recomputing only a subset of positions at each step, chosen either on a fixed schedule or by heuristic signals of change.

摘要：扩散大语言模型（dLLMs）通过迭代掩码去噪实现灵活生成，但由于在去噪步骤中需要重复计算标记表示，导致了高昂的推理成本。现有的基于缓存的方法通过在去噪步骤间重用中间结果，并仅在每一步重计算部分位置（通过固定计划或启发式变化信号选择）来降低此成本。

Our trajectory experiments instead point to decoding trajectory as the more relevant quantity. Cached generation remains accurate when it stays close to the full-denoising trajectory, with errors concentrated around unmask-order deviations and reduced as longer reference prefixes are recovered. This suggests a more direct principle: limiting representation error helps, but keeping generation on the full-denoising trajectory is a more direct route to accurate decoding.

我们的轨迹实验表明，解码轨迹才是更相关的量。当缓存生成保持在接近全去噪轨迹的状态时，其准确性较高；误差主要集中在去掩码顺序的偏差上，并随着更长参考前缀的恢复而减少。这提出了一个更直接的原则：限制表示误差固然有帮助，但保持生成过程处于全去噪轨迹上才是实现准确解码的更直接途径。

We therefore formulate efficient dLLM inference as trajectory-guided querying. Rather than broadly repairing representation error, we use full-denoising trajectories to supervise which unresolved positions should receive limited fresh computation. Based on this formulation, we introduce TRACEQUERY, a lightweight sparse decoding framework that predicts a small set of near-future trajectory-relevant positions, queries only those candidates, and lets the backbone select which one to commit from their fresh predictions.

因此，我们将高效的 dLLM 推理表述为轨迹引导查询。我们不再广泛地修复表示误差，而是利用全去噪轨迹来监督哪些未解决的位置应进行有限的新计算。基于这一表述，我们引入了 TRACEQUERY，这是一个轻量级的稀疏解码框架，它预测一小部分近期与轨迹相关的位置，仅查询这些候选位置，并让主干网络从其最新预测中选择要提交的内容。

TRACEQUERY delivers the strongest quality-computation trade-off among cache-based approaches, sustaining high accuracy with substantially fewer fresh token computations, and generalizes across tasks, model variants, and families.

TRACEQUERY 在基于缓存的方法中提供了最强的质量与计算权衡，在显著减少新标记计算量的同时保持了高准确性，并可推广至各种任务、模型变体和模型家族。