Quantization-Aware Healing: a compressed, 4-bit model that outperforms its full-precision original
Quantization-Aware Healing: a compressed, 4-bit model that outperforms its full-precision original
量化感知修复（QAH）：一种超越其全精度原始模型的压缩 4-bit 模型
Making a large language model smaller almost always comes with a cost. The now-standard recipe for efficient deployment is to compress the architecture first, cutting the parameter count by removing layers, heads, or neurons, and then quantize the remaining weights down to 4 bits to shrink memory and compute further. Both steps save a lot, but together they systematically degrade the capabilities people actually care about: reasoning, mathematical problem-solving, and code generation. 缩小大语言模型的规模几乎总是需要付出代价。目前高效部署的标准流程是先压缩架构，通过移除层、注意力头或神经元来减少参数量，然后将剩余权重压缩至 4-bit，以进一步降低内存和计算需求。这两个步骤虽然节省了大量资源，但结合起来会系统性地削弱用户真正关心的能力：推理、数学问题解决和代码生成。
Because of this, serious deployment pipelines add a recovery step, usually called healing, before the model goes into production. Recent open-weight releases such as gpt-oss, NVIDIA’s Nemotron family, and our own Hypernova 60B all rely on some version of this compress-then-heal approach. 因此，严谨的部署流程会在模型投入生产前增加一个恢复步骤，通常称为“修复”（healing）。近期发布的开源权重模型，如 gpt-oss、NVIDIA 的 Nemotron 系列以及我们自己的 Hypernova 60B，都依赖于这种“先压缩、后修复”方法的某种变体。
Our latest paper, Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMs, asks a question that the field has mostly left open: once a model has already been through structural compression, not just quantization, how well does that recovery step actually work, and what is the right way to do it? 我们最新的论文《量化感知修复：恢复压缩 4-bit 大语言模型的实用配方》（Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMs）提出了一个该领域尚未解决的问题：当模型不仅经过了量化，还经历了结构性压缩后，修复步骤的效果究竟如何？正确的做法又是什么？
We introduce Quantization-Aware Healing (QAH), and applied to a GPT-OSS 120B model compressed to 60B parameters and quantized to MXFP4, it produces a model that beats its own full-precision (bfloat16) version on 7 of 9 benchmarks. The 4-bit model ends up smaller, cheaper to run, and more accurate than the checkpoint it was quantized from. This inverts the usual relationship between a 4-bit model and the 16-bit model it came from. 我们引入了“量化感知修复”（QAH）。将其应用于一个从 120B 压缩至 60B 参数并量化为 MXFP4 的 GPT-OSS 模型时，该模型在 9 项基准测试中的 7 项表现超过了其自身的全精度（bfloat16）版本。最终得到的 4-bit 模型不仅体积更小、运行成本更低，而且比其量化前的检查点更准确。这颠覆了 4-bit 模型与其原始 16-bit 模型之间的传统关系。
Why the usual healing methods fall short here
为什么常规的修复方法在此处失效
Most efficiency pipelines follow the same three steps: compress the architecture, quantize the compressed weights, then heal the damage. The difference between methods is entirely in that last step. The dominant healing recipe is quantization-aware training (QAT). It inserts fake-quantization operators into the forward pass and keeps fine-tuning the model on a task loss, so the weights learn to tolerate the low-precision representation. 大多数效率优化流程遵循相同的三个步骤：压缩架构、量化压缩后的权重，然后修复损伤。不同方法之间的差异完全在于最后一步。目前主流的修复配方是量化感知训练（QAT）。它在正向传播中插入伪量化算子，并持续在任务损失函数上微调模型，使权重学会适应低精度表示。
In practice this means re-running an already expensive multi-stage post-training process, supervised fine-tuning, RLHF, agentic tuning, through a noisier, lower-precision forward pass. It is costly, and as our results show, it can also become unstable if training continues too long past its best point. 在实践中，这意味着需要通过一个噪声更大、精度更低的正向传播过程，重新运行已经非常昂贵的多阶段后训练流程（包括监督微调、RLHF 和智能体调优）。这不仅成本高昂，而且正如我们的结果所示，如果训练在最佳点之后持续时间过长，还可能变得不稳定。
An alternative, quantization-aware distillation (QAD), avoids re-running that history. Instead of a task loss, it distills a frozen full-precision teacher directly into the quantized student through a KL-divergence loss on the output logits. This works well when the only change is quantization, because a genuine full-precision version of the exact same model exists to act as teacher. 另一种替代方案是量化感知蒸馏（QAD），它避免了重新运行上述历史流程。它不使用任务损失，而是通过输出 Logits 上的 KL 散度损失，将冻结的全精度教师模型直接蒸馏到量化后的学生模型中。当唯一的改变仅是量化时，这种方法效果很好，因为存在一个完全相同的全精度模型版本可以作为教师。
But once a model has gone through structural compression, fewer layers, heads, or neurons, and not just fewer bits, that assumption breaks. There is no independently trained full-precision version of the smaller architecture. The only candidate teacher is the recovered bfloat16 checkpoint, which is itself a distilled approximation of the original model. Distilling from it anchors the quantized student to a degraded target and caps its accuracy at that recovered checkpoint’s own ceiling. 但一旦模型经历了结构性压缩（即减少了层数、注意力头或神经元，而不仅仅是减少位数），这一假设就不再成立。因为不存在一个针对较小架构独立训练的全精度版本。唯一的候选教师是恢复后的 bfloat16 检查点，而它本身只是原始模型的一个蒸馏近似。从它进行蒸馏会将量化后的学生模型锚定在一个退化的目标上，并将其准确率限制在该检查点的上限内。
So the question of how to heal a model that has been both structurally compressed and quantized was, until now, genuinely open. 因此，如何修复一个既经过结构性压缩又经过量化的模型，在目前为止仍然是一个悬而未决的问题。
Our approach
我们的方法
QAH removes that ceiling with one change: it distills directly from the original, pre-compression model rather than from the recovered one. Teacher and student do not even share an architecture. The teacher is full-size and full-precision, the student is half the size and running in MXFP4. Because a teacher’s output distribution is architecture-agnostic, nothing about the size or shape mismatch prevents the transfer. QAH 通过一个改变消除了这一上限：它直接从原始的、压缩前的模型进行蒸馏，而不是从恢复后的模型进行蒸馏。教师模型和学生模型甚至不需要共享架构。教师模型是全尺寸、全精度的，而学生模型是其一半大小并运行在 MXFP4 下。由于教师模型的输出分布与架构无关，尺寸或形状的不匹配不会阻碍知识迁移。
The student never sees hard labels, only the teacher’s output distribution, matched through KL divergence on the logits. This reframes what the quantization stage is doing. Under QAH it is no longer a lossy postprocessing step applied after healing is finished. It is a second, full pass of distillation against the original teacher, supervision that the bfloat16 checkpoint never received. 学生模型从不接触硬标签（hard labels），只通过 Logits 上的 KL 散度匹配教师模型的输出分布。这重新定义了量化阶段的作用。在 QAH 下，它不再是修复完成后进行的有损后处理步骤，而是针对原始教师模型的第二次完整蒸馏过程，这是一种 bfloat16 检查点从未获得过的监督。
The 4-bit student is not compensating for information lost to quantization; it is picking up information the earlier recovery stage did not have the time or data to transfer. There is also a stability benefit that falls out of the loss itself. Because KL distillation ties the student to a fixed teacher distribution, once the student catches up there is no further pressure for it to drift. 4-bit 学生模型并非在补偿量化带来的信息损失，而是在获取早期恢复阶段因时间和数据限制未能传递的信息。这种方法还带来了损失函数本身所具备的稳定性优势。由于 KL 蒸馏将学生模型绑定到一个固定的教师分布上，一旦学生模型追赶上来，它就不会再有偏离的压力。
A cross-entropy task loss, by contrast, keeps pushing the student toward hard labels indefinitely. That difference turns out to matter for both accuracy and training stability, as the comparison below shows. 相比之下，交叉熵任务损失会不断地将学生模型推向硬标签。正如以下对比所示，这种差异对于准确率和训练稳定性都至关重要。
To make QAH work at long context, where the healing corpus includes documents up to 32k tokens, we reuse the memory-efficient chunked KL-divergence loss from our companion paper on efficient distillation. That loss computes the KL one slice of the sequence at a time and never materializes the full vocabulary-by-sequence grid, which is what makes 32k-token healing fit inside a fixed GPU memory budget. 为了使 QAH 在长上下文（修复语料库包含长达 32k token 的文档）下工作，我们复用了我们在另一篇关于高效蒸馏的论文中提出的内存高效分块 KL 散度损失。该损失函数每次计算序列的一个片段，从不生成完整的“词表 x 序列”网格，这使得 32k token 的修复过程能够适应固定的 GPU 内存预算。
QAH overview. After structural compression and quantization, capabilities drop sharply. QAH distills from the original model, a frozen teacher whose logits are precomputed offline, rather than from the recovered checkpoint. QAH 概览。在结构性压缩和量化后，模型能力会急剧下降。QAH 从原始模型（一个 Logits 离线预计算的冻结教师模型）进行蒸馏，而不是从恢复后的检查点进行蒸馏。