PRQuant: Permutation Residual Quantization for Low-Overhead Inference
PRQuant: Permutation Residual Quantization for Low-Overhead Inference
PRQuant：用于低开销推理的置换残差量化
Abstract: Accuracy of Low-bit quantization of linear layers is often dominated by a small number of outliers. Although existing methods, such as smoothing, rotation, or residual-based approaches, may mitigate this problem, they often introduce new accuracy bottlenecks to weights. Besides, most of these techniques are implemented as online approaches, which can result in heavy execution overheads.
摘要： 线性层的低位量化精度往往受少数异常值的影响。尽管现有的方法（如平滑、旋转或基于残差的方法）可以缓解这一问题，但它们往往会给权重带来新的精度瓶颈。此外，这些技术大多以在线方式实现，这可能导致沉重的执行开销。
To address the afore-mentioned issues, We propose PRQuant (Permutation Residual Quantization), a training-free and low-overhead framework that combines channel reorganization with static weight-side residual compensation. After AWQ-style scaling, PRQuant identifies the input channels that contribute most to weight quantization error, permutes them into contiguous tail blocks, and constructs their residual weight sub-tensors offline.
为了解决上述问题，我们提出了 PRQuant（置换残差量化），这是一个无需训练且低开销的框架，它将通道重组与静态权重侧残差补偿相结合。在进行 AWQ 风格的缩放后，PRQuant 会识别出对权重量化误差贡献最大的输入通道，将它们置换为连续的尾部块，并在离线状态下构建其残差权重子张量。
During inference, this contiguous structure enables the activation side to use tail blocks seamlessly without the expensive online gathering operation, and turns scattered residual compensation into a regular tail-augmented GEMM, substantially reducing latency.
在推理过程中，这种连续结构使激活侧能够无缝使用尾部块，而无需昂贵的在线收集操作，并将分散的残差补偿转化为规则的尾部增强 GEMM（通用矩阵乘法），从而大幅降低了延迟。
Experiments demonstrate that PRQuant effectively reduces down-projection reconstruction error. Ablation studies confirm that smoothing and residual compensation are the primary drivers of numerical improvement, while permutation provides a consistent marginal numerical benefit and, more importantly, enables a hardware-friendly contiguous layout that eliminates dynamic gathering overhead.
实验表明，PRQuant 有效地降低了下投影（down-projection）的重构误差。消融研究证实，平滑和残差补偿是数值提升的主要驱动力，而置换则提供了持续的边际数值收益，更重要的是，它实现了对硬件友好的连续布局，消除了动态收集开销。
Overall, PRQuant outperforms default MXFP4 and the evaluated PTQ baselines in average accuracy across five downstream benchmarks, improving over MXFP4 by 1.24 and 0.55 on Qwen3-4B-Instruct-2507 and Qwen3-30B-A3B-Instruct-2507, respectively.
总体而言，PRQuant 在五个下游基准测试中的平均准确率均优于默认的 MXFP4 和所评估的 PTQ 基线，在 Qwen3-4B-Instruct-2507 和 Qwen3-30B-A3B-Instruct-2507 上分别比 MXFP4 提升了 1.24 和 0.55 个百分点。