Quantization-Aware Healing Produces a 4-Bit Model That Outperforms Its Full-Precision Version

Quantization-Aware Healing Produces a 4-Bit Model That Outperforms Its Full-Precision Version

Reducing the size of a large language model usually involves two stages: structurally compressing the architecture by removing layers, attention heads, or neurons, then quantizing the remaining weights to lower precision. Although these steps reduce memory and compute requirements, they can also weaken reasoning, mathematical problem-solving, and code-generation capabilities.

Deployment pipelines therefore commonly add a recovery stage called healing. Recent open-weight releases, including gpt-oss, NVIDIA's Nemotron family, and Multiverse Computing's Hypernova 60B, use variations of a compress-then-heal process.

A paper titled Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMs examines how well healing works after both structural compression and quantization. It introduces Quantization-Aware Healing (QAH). Applied to a GPT-OSS 120B model compressed to 60B parameters and quantized to MXFP4, QAH produced a model that outperformed its full-precision bfloat16 counterpart on seven of nine benchmarks.

The resulting 4-bit model was smaller and less expensive to run, while also scoring higher than the checkpoint from which it was quantized.

Why conventional healing methods have limitations

Most efficiency pipelines use three stages:

1. Compress the architecture.
2. Quantize the compressed weights.
3. Heal the resulting model.

The main distinction between approaches is the loss function used during healing.

Quantization-aware training

Quantization-aware training (QAT) inserts fake-quantization operators into the forward pass and continues fine-tuning with a task loss. This requires rerunning an already expensive post-training process, such as supervised fine-tuning, RLHF, or agentic tuning, through a noisier, lower-precision computation path.

QAT is costly, and extended training can make it unstable. The model may reach its best performance and then deteriorate if training continues for too long.

Quantization-aware distillation

Quantization-aware distillation (QAD) instead transfers knowledge from a frozen, full-precision teacher to a quantized student. It uses a KL-divergence loss applied to the models' output logits.

This works well when quantization is the only change, because an exact full-precision version of the same architecture can serve as the teacher. Structural compression changes that assumption. Once layers, heads, or neurons have been removed, there is no independently trained full-precision model with the smaller architecture.

The available teacher is then the recovered bfloat16 checkpoint, which is already an approximation of the original model. Distilling from that checkpoint limits the quantized model to the recovered model's capabilities.

This left an open question: how should a model be healed after both structural compression and quantization?

The Quantization-Aware Healing approach

QAH addresses this limitation by distilling directly from the original, pre-compression model rather than from the recovered checkpoint.

The teacher and student have different architectures. The teacher is full-size and full-precision, while the student has half as many parameters and uses MXFP4. Because the teacher's output distribution does not depend on the architecture's size or shape, the mismatch does not prevent knowledge transfer. The student is trained against the teacher's output distribution through KL divergence rather than against hard labels.

This changes the role of quantization. Instead of being a lossy processing step performed after healing, quantization becomes another full distillation pass supervised by the original teacher. The 4-bit student can therefore learn information that the earlier recovery stage did not have enough time or data to transfer.

The loss also provides a stability benefit. KL distillation ties the student to a fixed teacher distribution. Once the student matches that distribution, it has little incentive to drift. A cross-entropy task loss continues pushing the model toward hard labels, even after it has reached its best performance.

For long-context healing, the training corpus contains documents of up to 32k tokens. QAH uses a memory-efficient chunked KL-divergence loss from related work on efficient distillation. The loss calculates KL divergence one sequence slice at a time instead of materializing the complete vocabulary-by-sequence grid, allowing 32k-token healing within a fixed GPU memory budget.

QAH uses the original model as a frozen teacher. Its logits are precomputed offline rather than obtained from the recovered checkpoint.

Benchmark results

The researchers applied QAH to GPT-OSS 120B after compressing it to 60B parameters and recovering it in bfloat16. They then re-quantized the model to MXFP4 using QAH.

The comparison was made against the 60B model's bfloat16 checkpoint, which was the best available full-precision version of that architecture.

The QAH model exceeded its bfloat16 source on seven benchmarks. On MMLU-Pro and SciCode, it scored lower, with differences of less than one point and 1.4 points, respectively.

The largest gains occurred in areas often affected most by compression: long-context reasoning improved by 7.4 points on AA-LCR, while AIME 2025 improved by 5.6 points.

Compared with the original 120B teacher, the QAH model used half the parameter count and approximately one-quarter of the weight memory. It still exceeded the teacher on LiveCodeBench, scoring 66.5 compared with 66.0. On GPQA Diamond, it scored 67.4 compared with the teacher's 69.0. The largest remaining difference was on AA-LCR, an especially demanding long-context benchmark.

The 4-bit QAH model matched or exceeded its bfloat16 source on seven of nine benchmarks and exceeded the full-size teacher on LiveCodeBench.

QAH and QAT compared

To separate the effect of the loss function from other variables, the researchers compared QAH and QAT under matched conditions. They quantized GPT-OSS 9B to MXFP4 and tracked average performance across MMLU-Pro, LiveCodeBench, and GPQA Diamond as training progressed.

The methods reached similar peak scores: 54.9 for QAH and 54.6 for QAT. Their training behavior differed substantially:

• QAH reached its peak in approximately 100 steps, about seven times faster than QAT's 700 steps.
• QAH remained within about two points of its peak for the rest of training.
• QAT declined sharply after its peak, losing nearly 19 points by step 1,200.

This difference affects checkpoint selection. QAT requires careful early stopping using a held-out signal to avoid selecting a model that has already begun to degrade. A sufficiently trained QAH checkpoint is less sensitive to the exact stopping point because it does not drift in the same way.

The behavior is consistent with the objectives. KL distillation against a frozen teacher gives the student little reason to change after matching the teacher. Cross-entropy continues optimizing against hard labels and can eventually damage capabilities inherited from the original model.

QAH reached 54.9 in approximately 100 steps and remained stable. QAT reached 54.6 around step 700, then lost nearly 19 points by step 1,200.

Deployment implications

The accuracy results accompany the memory and compute reductions that motivate compression.

At 4-bit precision, the QAH model uses approximately four times less weight memory than the bfloat16 student. Because it has half the parameter count of the 120B teacher, it also roughly halves compute per token. This allows the model to run on smaller hardware.

For model families distributed in bfloat16 rather than 4-bit form, combining parameter reduction with precision reduction would result in closer to eight times less compute per token.

These results show that a compressed 4-bit model does not necessarily have to be a less accurate version of its full-precision counterpart. With QAH, the model can be smaller and less expensive to serve while scoring higher on most of the evaluated benchmarks. The approach also reaches its peak in a fraction of the training steps required by QAT.

QAH uses the original pre-compression model as its teacher and treats quantization as an additional opportunity for knowledge transfer rather than only as a source of information loss.