Optimizing Language Model Inference: Techniques from Quantization to Speculative Decoding Part 1

Introduction

In discussions about AI, training large models often takes the spotlight, involving massive GPU clusters, extensive datasets, and substantial financial investments. However, once a model is trained, it must execute thousands of times daily for users with specific latency and budget constraints. This process is known as inference, a critical phase where most engineering challenges arise. When a user interacts with a chatbot and expects an instantaneous reply, the model must generate tokens rapidly, manage memory efficiently, and do so cost-effectively. Therefore, optimizing inference is paramount.

Recent advancements have introduced numerous strategies to make large language models (LLMs) faster, more efficient, and cost-effective without significant accuracy loss. These strategies can either alter the model's internal structure or optimize its deployment and execution. The most successful deployments integrate several of these strategies.

This article, the first in a two-part series, will explore five primary techniques for optimizing LLM inference:

• Quantization: Reduces the precision of model weights, slimming down models from 32 bits per number to as few as 8, 4, or even fewer, with minimal quality degradation.
• Pruning: Eliminates non-essential components, such as redundant attention heads and inactive layers, to maintain capability while reducing size.
• Knowledge Distillation: Transfers knowledge from a large model to a smaller one, teaching the smaller model to mimic the larger one rather than learning from scratch.
• KV Caching: Addresses the computation-heavy task of autoregressive generation by storing and reusing intermediate computations.
• Speculative Decoding: Utilizes a small, fast model to predict multiple tokens, with the larger model verifying them in parallel, effectively overcoming sequential constraints.

Each technique addresses distinct challenges in inference optimization, and understanding their interplay is as crucial as comprehending each one individually.

In this first part, the focus will be on quantization and pruning, exploring how these techniques streamline large models for efficient real-world application.

Key Takeaways

• Quantization reduces model weight precision, making LLMs more compact, faster, and cheaper to run without greatly affecting quality.
• Pruning removes less significant weights or connections, enhancing efficiency and reducing memory usage.
• Techniques like GPTQ, AWQ, and LLM.int8() enable quantization of pretrained models without necessitating full retraining.
• Combining pruning and quantization can significantly boost performance, particularly for edge devices and cost-conscious deployments.
• Moderate optimization usually retains model quality close to the original, but aggressive compression may impair accuracy and reasoning capabilities.
• Calibration datasets assist quantization methods in understanding model behavior in real workloads, typically requiring minimal sample data.
• As organizations seek efficient deployment of large models without expensive GPU infrastructure, optimization becomes essential.

Quantization — Shrinking the Numbers Without Losing the Meaning

Quantization is a strategy to make AI models smaller, faster, and less expensive by reducing the precision of model weights and activations. While training is often highlighted, the question arises: does a neural network truly need 32 bits for number storage? Typically, the answer is no, paving the way for quantization.

What Is a Weight, Really?

A neural network learns by adjusting numerous numerical values, known as weights, which represent the model's knowledge. During training, these weights are continually updated to enhance prediction accuracy.

Weights are stored as floating-point numbers, which are a method to represent numbers with decimals. Computers use bits to store these numbers, with common formats in deep learning being:

| Format | Size | |--------|----------| | FP32 | 32 bits | | FP16 | 16 bits | | BF16 | 16 bits | | INT8 | 8 bits |

The Problem With FP32 is its high accuracy but substantial cost, leading to increased memory usage, slower data transfer, higher power consumption, and costly inference.

Neural networks often do not require extremely precise numbers. For example, a number like 0.123456789 can be approximated as 0.12 without significantly affecting the model's output. This idea forms the basis of quantization.

Quantization converts large, high-precision numbers like FP32 into smaller, efficient formats such as FP16, BF16, INT8, or even INT4. The goal is to minimize memory usage and enhance inference speed without substantially affecting output quality.

Large language models contain billions of parameters, with each represented as a number. In FP32 format, each parameter consumes 32 bits of memory. Quantization reduces these numbers' size, enabling the model to run more efficiently on smaller GPUs, consumer hardware, edge devices, and large-scale inference servers.

For example, a 7 billion parameter model in FP32 might require 28 GB of memory, but when quantized to INT8, it might only need about 7 GB. This reduction significantly lowers infrastructure costs and enhances scalability.

The core idea of quantization is that neural networks can function well with approximate decimal values. During inference, a model can often work with such approximations without noticeable quality loss, converting floating-point numbers into lower-precision formats while retaining the model's overall meaning and behavior.

PTQ Example in PyTorch

import torch
import torch.nn as nn
import torch.quantization

## Simple neural network
class SimpleModel(nn.Module):
    def __init__(self):
        super(SimpleModel, self).__init__()

        self.fc1 = nn.Linear(10, 32)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(32, 2)

    def forward(self, x):
        x = self.fc1(x)
        x = self.relu(x)
        x = self.fc2(x)
        return x

## Create model
model = SimpleModel()

## Set model to evaluation mode
model.eval()

## Specify quantization configuration
model.qconfig = torch.quantization.get_default_qconfig("fbgemm")

## Prepare the model for static quantization
torch.quantization.prepare(model, inplace=True)

## Calibration step
## Run some sample data through the model
sample_input = torch.randn(100, 10)

with torch.no_grad():
    model(sample_input)

## Convert model to INT8 quantized version
torch.quantization.convert(model, inplace=True)

## Test inference
test_input = torch.randn(1, 10)

with torch.no_grad():
    output = model(test_input)

print(output)

Dynamic Quantization, where weights are pre-quantized, but activations are quantized during runtime, is another popular method. It is simple to implement and works well on CPUs. PyTorch supports dynamic quantization with minimal code, making it a common choice for production APIs and lightweight deployments.

quantized_model = torch.quantization.quantize_dynamic(
    model,
    {nn.Linear},
    dtype=torch.qint8
)

Static Quantization, meanwhile, quantizes both weights and activations before inference begins, requiring a calibration dataset to determine activation value ranges for real usage. This often provides better performance and lower latency than dynamic quantization, making it suitable for mobile AI systems, embedded devices, and optimized inference engines.

For scenarios demanding high accuracy, Quantization-Aware Training (QAT) is used. In QAT, the model simulates quantization effects during training, allowing it to adapt to lower precision while optimizing weights. This approach generally maintains accuracy better than post-training quantization, especially for aggressive formats like INT4, and is common in high-performance environments where accuracy loss is unacceptable.

Advanced quantization techniques, such as GPTQ, AWQ, and GGUF formats, are increasingly used in modern LLM serving systems. GPTQ, or Generalized Post-Training Quantization, reduces quantization error layer by layer, enabling large models to run efficiently in 4-bit precision with minimal quality loss. AWQ, or Activation-Aware Weight Quantization, identifies important weights for preserving activations during quantization. These methods are now standard for deploying models on consumer GPUs.

Quantization is also integral to modern inference frameworks, such as TensorRT, ONNX Runtime, vLLM, and llama.cpp. These systems combine quantization with other optimizations like kernel fusion, speculative decoding, and KV caching to maximize throughput and minimize latency for real-time AI applications.

In real-world deployments, quantization allows AI systems to scale economically. Cloud providers use quantized models to serve numerous concurrent users while reducing GPU costs. Edge AI systems use quantization for running computer vision models on phones, drones, and IoT devices. Chatbots and RAG pipelines rely on quantized LLMs to reduce inference latency and enhance response speed. Without quantization, many AI products would be prohibitively expensive to operate at scale.

However, quantization involves trade-offs. Lower precision formats reduce memory usage and enhance speed but introduce approximation errors. Excessive quantization can lead to hallucinations, reduced reasoning quality, or unstable outputs. Therefore, selecting the correct precision level is crucial. Many production systems now employ mixed precision strategies, where sensitive layers remain in higher precision while less critical layers are aggressively quantized.

The success of quantization lies in the realization that neural networks can tolerate small numerical approximations. By carefully reducing numbers without losing their meaning, quantization makes modern AI practical, scalable, and affordable.

Pruning — Teaching a Model to Do More with Less

While quantization reduces the size of numbers within a model, pruning takes a more aggressive approach by removing entire model components. Pruning is akin to practices in decision trees, where certain branches or decisions are eliminated to reduce overfitting.

The idea is straightforward: not every parameter in a neural network contributes equally after training. Some attention heads and layers might be nearly redundant, and certain weights may contribute insignificantly to the output. Pruning identifies and removes these less impactful parts, resulting in a smaller, faster model with minimal capability loss.

The Lottery Ticket Hypothesis

In 2018, a pivotal finding revealed that within every large, dense neural network lies a smaller subnetwork, or "winning lottery ticket," that can achieve the same accuracy if trained independently from the start.

This discovery suggested that large models might be over-parameterized by design for easier optimization during training. Once training is complete, the important parts of the model can be isolated and retained.

This led to a shift in focus from whether parameters can be removed to how to identify which ones to remove.

Unstructured vs. Structured Pruning

Pruning can be unstructured or structured, with each type offering distinct benefits.

Unstructured pruning zeros out individual weights regardless of their position. While this achieves excellent compression ratios and quality preservation, the resulting irregular sparsity may not lead to faster inference on standard hardware without specialized sparse kernels.

Structured pruning, on the other hand, removes entire structural units like full attention heads or entire layers, resulting in a smaller model shape. This approach offers real throughput gains with standard hardware but may impact quality more than unstructured pruning.

For faster inference, structured pruning is recommended. For smaller model files with sparse compute, unstructured pruning provides better quality-compression trade-offs.

Magnitude Pruning

The simplest pruning method is to remove weights with the smallest absolute values, as they contribute minimally to the output. Magnitude pruning works best iteratively, pruning small fractions of weights and fine-tuning the model to allow for recovery.

The limitation of magnitude pruning is that weight magnitude is an indirect measure of importance. More sophisticated methods account for this by considering other factors.

Attention Head Pruning

Transformers utilize multi-head attention to focus on different input aspects simultaneously. However, studies have shown that many attention heads are redundant, with some being nearly identical or barely activated. Removing such heads can reduce compute costs significantly with minimal quality loss.

Layer Dropping and Depth Pruning

Structured pruning can remove entire transformer layers to reduce model size. However, determining which layers to drop is crucial, as some layers contribute more significantly to the model's performance.

Layer dropping can also be dynamic, allowing models to exit early during inference when confidence is high, improving average-case latency without altering maximum capability.

SparseGPT and Wanda

For large models, retraining or layer-wise optimization can be expensive. SparseGPT and Wanda offer one-shot pruning methods that require no retraining. SparseGPT adapts second-order Hessian frameworks for pruning, while Wanda considers weight magnitude and activation importance to decide which weights to remove.

Both methods offer efficient pruning of large models with minimal quality loss, making them valuable for reducing GPU costs, edge deployment, faster inference, and lower VRAM usage.

The Hardware Reality

GPUs are optimized for dense tensor operations, so they may not benefit from sparsity unless specific patterns are followed. CPU inference, however, can better utilize sparse models by reducing data load, improving performance.

FAQs

Do I need to retrain my model after quantization?

Typically, retraining is not necessary. Methods like GPTQ, AWQ, and LLM.int8() can quantize pretrained models using small sample datasets. Retraining may be needed for extremely low precision or tasks requiring high accuracy.

What’s the difference between GPTQ and AWQ — which should I use?

Both are quantization methods for making LLMs smaller and faster. GPTQ focuses on model size reduction while maintaining accuracy and is widely supported. AWQ emphasizes preserving quality by considering important activations, especially for chat and instruction-following models.

Can I apply both pruning and quantization to the same model?

Yes, combining pruning and quantization is common in optimized LLM pipelines. Pruning removes unnecessary parts, while quantization reduces the precision of remaining weights, enhancing memory efficiency and inference speed.

Will users notice a difference with a quantized or pruned model?

Generally, moderate optimization does not significantly affect user experience. INT8 quantization often appears identical to the original model, while heavy pruning might affect accuracy or increase repetitiveness.

How do I know which layers are safe to prune?

Modern pruning methods analyze weight importance, activation patterns, and layer sensitivity to determine which weights are less important. Attention and feed-forward layers often have removable weights, while early and final layers require careful pruning.

Conclusion: The First Half

LLM inference involves various techniques, and this article covers two key optimization strategies: quantization and pruning. Excess in models can be numerical, in the form of unnecessarily precise weights, or structural, in redundant components. By addressing both, methods like AWQ, GPTQ, Wanda, and SparseGPT offer efficient optimization.

In the second part of this series, the focus will shift to techniques at the generation level, including Knowledge Distillation, KV Caching, and Speculative Decoding, which further enhance model efficiency and performance.