Lossy Compressive Text Autoencoders

查看原文 / View Original 编辑 / Edit

Computer Science > Computation and Language arXiv:2610.10738 (cs) [Submitted on 7 Oct 2026] Title:Lossy Compressive Text Autoencoders Authors:Vinko Sabolčec, Angelos Katharopoulos, David Grangier View a PDF of the paper titled Lossy Compressive Text Autoencoders, by Vinko Sabol\v{c}ec and 2 other authors View PDF HTML (experimental)

计算机科学 > 计算与语言 arXiv:2610.10738 (cs) [提交于 2026 年 10 月 7 日] 标题：有损压缩文本自动编码器作者：Vinko Sabolčec, Angelos Katharopoulos, David Grangier 查看标题为《有损压缩文本自动编码器》的论文 PDF，作者为 Vinko Sabolčec 及其他 2 位作者查看 PDF HTML（实验性）

Abstract:Our work explores learning a compressed latent representation of text, at the intersection of data compression and representation learning. We propose an autoencoder architecture that performs residual downscaling and upscaling of hidden representations along the time axis, with a residual low-dimension discrete bottleneck. We analyze our approach for different quantization methods, training objectives, and datasets.

摘要：我们的工作探索了文本压缩潜在表示的学习，处于数据压缩和表示学习的交叉领域。我们提出了一种自动编码器架构，该架构沿时间轴对隐藏表示执行残差降采样和升采样，并具有残差低维离散瓶颈。我们针对不同的量化方法、训练目标和数据集分析了我们的方法。

For different levels of compression, we evaluate the similarity between the original and reconstructed text both at the surface-level (BLEU) and at the semantic-level (LLM-based judge). Additionally, we evaluate our models on downstream question-answering and semantic text similarity benchmarks. Our approach results in compressed representations which are on par with lossless text compression algorithms at 2.24 bits per byte on web text data, while having good reconstruction and downstream task performance.

针对不同的压缩级别，我们从表面层面（BLEU）和语义层面（基于大语言模型的评估）评估了原始文本与重构文本之间的相似度。此外，我们在下游问答和语义文本相似度基准上评估了我们的模型。我们的方法产生的压缩表示在网络文本数据上达到每字节 2.24 比特，与无损文本压缩算法相当，同时具备良好的重构和下游任务性能。