Neuromorphic Diffusion Language Models: Addressing Compute and Memory Bottlenecks via Sparsity and Block Denoising

神经形态扩散语言模型：通过稀疏性和块去噪解决计算与内存瓶颈

Abstract: Autoregressive (AR) large language models (LLMs) are inherently inefficient at inference time because each generated token requires accessing the full set of model parameters, leading to low operational intensity and high energy consumption. 摘要： 自回归（AR）大语言模型（LLM）在推理阶段本质上效率低下，因为每个生成标记（token）都需要访问全套模型参数，这导致了较低的运算强度和较高的能耗。

Masked diffusion language models (MDLMs) partially address this limitation for memory-bound settings by allowing multiple tokens to be generated per parameter access. 掩码扩散语言模型（MDLM）通过允许每次参数访问生成多个标记，在内存受限的场景下部分解决了这一局限性。

In order to further enhance inference efficiency on modern platforms with extensive in-chip memory, this work proposes neuromorphic MDLMs (N-MDLMs), which integrate block diffusion with spike-based neuromorphic computation to jointly improve throughput and energy efficiency. 为了在具有大容量片上内存的现代平台上进一步提高推理效率，本文提出了神经形态 MDLM（N-MDLMs），它将块扩散与基于脉冲的神经形态计算相结合，以共同提升吞吐量和能效。

While block diffusion increases token throughput by producing multiple tokens per parameter access, spike-induced sparsity reduces effective parameter traffic and computations by skipping inactive channels. 虽然块扩散通过每次参数访问生成多个标记来增加标记吞吐量，但脉冲诱导的稀疏性通过跳过非活动通道，减少了有效的参数流量和计算量。

To analyze the synergistic effect of sparsity and diffusion, we develop a token-level roofline-inspired model that captures the combined impact of block-parallel generation and spike sparsity on decoding efficiency. 为了分析稀疏性和扩散的协同效应，我们开发了一种受“屋顶线模型”（roofline model）启发的标记级模型，用以捕捉块并行生成和脉冲稀疏性对解码效率的综合影响。

Experimental results on translation tasks show that, thanks to spike-induced sparsity, N-MDLMs achieve substantial improvements in energy efficiency and throughput even in compute-bound platforms for which MDLMs would fail to improve over AR-LLMs. 在翻译任务上的实验结果表明，得益于脉冲诱导的稀疏性，N-MDLM 在计算受限的平台上也能实现能效和吞吐量的显著提升，而这些平台通常是 MDLM 无法超越 AR-LLM 的场景。