AutoIndexer: Flexible-Order Decoding by Training Causal Models on Chains of Edits

查看原文 / View Original 编辑 / Edit

Computer Science > Computation and Language arXiv:2610.12545 (cs) [Submitted on 8 Oct 2026] Title:AutoIndexer: Flexible-Order Decoding by Training Causal Models on Chains of Edits Authors:Shu Ishida, Aliasghar Khani, Tianyu Zhang, The Cong Luong, James Seale Smith, Adam Gaier

计算机科学 > 计算与语言 arXiv:2610.12545 (cs) [提交于 2026 年 10 月 8 日] 标题：AutoIndexer：通过在编辑链上训练因果模型实现灵活顺序解码作者：Shu Ishida, Aliasghar Khani, Tianyu Zhang, The Cong Luong, James Seale Smith, Adam Gaier

Abstract: Autoregressive decoding is less suited to non-causally ordered data, e.g., cross-references or future dependencies, since their outputs cannot be revised post hoc, unlike iterative human reasoning. Diffusion LLMs partially mitigate this limitation but still cannot flexibly insert or remove arbitrary numbers of tokens from a pre-generated context.

摘要：自回归解码不太适合非因果顺序的数据（例如交叉引用或未来依赖关系），因为与人类的迭代推理不同，其输出无法事后修改。扩散大语言模型（Diffusion LLMs）部分缓解了这一局限性，但仍然无法从预生成的上下文中灵活地插入或删除任意数量的标记（token）。

We design AutoIndexer as a generative model with a novel training and decoding mechanism that simulates a Chain of Edits, allowing the model to flexibly modify its prior outputs in hindsight via token insertions, substitutions, and deletions. The model learns to predict marker tokens for opening and closing edits, simultaneously moving a cursor to positions in the existing token sequence, analogous to a text editor.

我们设计了 AutoIndexer 作为一种生成模型，它具有一种模拟“编辑链”的新型训练和解码机制，允许模型通过标记插入、替换和删除，在事后灵活地修改其先前的输出。该模型学习预测用于开启和关闭编辑的标记符号，同时将光标移动到现有标记序列中的位置，类似于文本编辑器。

Training sequences are obtained by perturbing ground-truth labels with inverse chained edit operations, and the model learns to edit by undoing those inverses. The perturbation training allows AutoIndexer to be robust to erroneous context and adaptable to editable tasks such as code correction.

训练序列是通过使用逆向链式编辑操作扰动真实标签获得的，模型通过撤销这些逆向操作来学习编辑。这种扰动训练使 AutoIndexer 能够对错误的上下文保持稳健，并适应代码纠错等编辑任务。

We demonstrate that our method exhibits post-generation context-editing capabilities that most autoregressive and diffusion models lack without compromising performance on standard LLM benchmarks. The code is available at this https URL.

我们证明，我们的方法展现出了大多数自回归和扩散模型所缺乏的事后上下文编辑能力，且不会影响在标准大语言模型基准测试上的性能。代码可在该网址获取。