Exploring the Design Space of Permutation Diffusion LLMs

查看原文 / View Original 编辑 / Edit

Computer Science > Computation and Language arXiv:2610.20635 (cs) [Submitted on 1 Oct 2026] Title: Exploring the Design Space of Permutation Diffusion LLMs Authors: Zhe Xu, Mingfu Liang, Kavosh Asadi, Jean-Pascal Billaud, Tianyi Chen, Yanhui Guo, Christina Huang, Baekjin Kim, Jimmy Kim, Yufei Li, Daqi Sun, Jiachen Sun, Dian Yu, Zaiwei Zhang, Kaushik Rangadurai, Frank Shyu, Sandeep Pandey, Xi Liu, Luke Simon.

计算机科学 > 计算与语言 arXiv:2610.20635 (cs) [提交于 2026 年 10 月 1 日] 标题：探索排列扩散大语言模型的设计空间作者：Zhe Xu, Mingfu Liang, Kavosh Asadi, Jean-Pascal Billaud, Tianyi Chen, Yanhui Guo, Christina Huang, Baekjin Kim, Jimmy Kim, Yufei Li, Daqi Sun, Jiachen Sun, Dian Yu, Zaiwei Zhang, Kaushik Rangadurai, Frank Shyu, Sandeep Pandey, Xi Liu, Luke Simon。

Abstract: Permutation generation with language models requires both text-conditioned ranking and a global one-to-one assignment between candidates and output positions. We study this problem through three diffusion-LLM parameterizations: hard-assignment prediction over shuffled candidate-index sequences, soft-assignment prediction over doubly stochastic matrices, and direct ordering-score prediction followed by sorting.

摘要：使用语言模型生成排列需要文本条件排序以及候选对象与输出位置之间的全局一一对应分配。我们通过三种扩散大语言模型参数化方法研究了这一问题：基于打乱的候选索引序列的硬分配预测、基于双随机矩阵的软分配预测，以及直接排序分数预测后进行排序。

The three families place permutation constraints at different stages of prediction and decoding and therefore support different training objectives and sampling procedures. We evaluate these designs on recommendation reranking across three Amazon domains with candidate sets of size 10 and 20.

这三个系列在预测和解码的不同阶段施加排列约束，因此支持不同的训练目标和采样过程。我们在三个亚马逊领域的推荐重排序任务上评估了这些设计，候选集大小分别为 10 和 20。

At one function evaluation, the hard- and soft-assignment variants achieve Recall@1 comparable to a constrained masked-diffusion baseline, whereas direct ordering-score prediction guarantees valid permutations without Hungarian assignment or Sinkhorn normalization. Additional function evaluations do not consistently improve ranking quality, and their effect depends strongly on the state representation and training objective.

在一次函数评估中，硬分配和软分配变体实现了与受约束掩码扩散基线相当的 Recall@1，而直接排序分数预测则无需匈牙利算法或 Sinkhorn 归一化即可保证有效的排列。额外的函数评估并不能持续提高排序质量，其效果在很大程度上取决于状态表示和训练目标。

These results show that corruption, prediction parameterization, objective, and decoder should be designed jointly for permutation diffusion with language-model denoisers.

这些结果表明，对于使用语言模型去噪器的排列扩散，损坏机制、预测参数化、目标函数和解码器应当进行联合设计。