Diffuse to Compress: Leveraging Diffusion LMs for Lossless Compression

压缩即扩散：利用扩散语言模型实现无损压缩

Abstract: We study the problem of lossless text compression, motivated by the rapid growth in the collection and storage of digital textual data - including plain text, source code, and structured formats such as XML - and by recent advances in neural language model-based compression.

摘要： 我们研究了无损文本压缩问题，其动机源于数字文本数据（包括纯文本、源代码以及 XML 等结构化格式）收集与存储的快速增长，以及近期基于神经语言模型压缩技术的进展。

In particular, recent LLM-based approaches, whether built on symbol-ranking pipelines or paired with a statistical compressor, have demonstrated compression ratios significantly superior to general-purpose compressors such as zstd, gzip, or bzip on text and code. However, these neural approaches suffer from severe throughput limitations, making them not yet practically usable.

特别是，近期基于大语言模型（LLM）的方法，无论是构建在符号排序流水线上，还是与统计压缩器配对，在文本和代码上的压缩比都显著优于 zstd、gzip 或 bzip 等通用压缩器。然而，这些神经方法存在严重的吞吐量限制，导致它们目前尚无法实际应用。

For the first time in the context of lossless neural text compression, we introduce Diffusion Language Models (DLMs) as an alternative inference paradigm to autoregressive LLM-based approaches. We argue that replacing autoregressive LLMs with DLMs within the same compression framework could overcome the throughput bottleneck caused by their one-symbol-per-step limitation.

在无损神经文本压缩领域，我们首次引入了扩散语言模型（DLMs），作为替代自回归 LLM 方法的一种推理范式。我们认为，在相同的压缩框架内用 DLMs 替换自回归 LLMs，可以克服因“每步仅处理一个符号”的限制所导致的吞吐量瓶颈。

However, achieving these improvements requires addressing algorithmic challenges introduced by applying DLMs to lossless compression, where the architecture allows the number and positions of symbols encoded at each forward pass to be decided independently.

然而，实现这些改进需要解决将 DLMs 应用于无损压缩时带来的算法挑战，因为该架构允许在每次前向传播中独立决定编码符号的数量和位置。

We design efficient and effective strategies to solve these challenges and evaluate them experimentally against LLM-based and general-purpose compressors on enwik8, a well-established textual benchmark. Our results show that the newly proposed DLM-based framework advances the state of the art in lossless text compression.

我们设计了高效且有效的策略来解决这些挑战，并在公认的文本基准测试 enwik8 上，将其与基于 LLM 的压缩器和通用压缩器进行了实验对比。结果表明，新提出的基于 DLM 的框架提升了无损文本压缩的行业领先水平。

Moreover, as DLMs are still a relatively young paradigm, recent advances toward increasingly capable and efficient models suggest substantial room for further improvements.

此外，由于 DLMs 仍是一个相对较新的范式，近期向更强大、更高效模型发展的进展表明，该领域仍有巨大的改进空间。