Post-Training Quantization

Why Does This Exist?

A 70 Billion parameter model trained in 16-bit precision (fp16) requires 140 GB of VRAM just to load the weights. That requires two $30,000 GPUs just to turn the model on.

During training, 16-bit precision is absolutely necessary. Gradients are tiny, and if you round them off, the model won't learn. But once the model is trained, the weights are frozen. They are just a lookup table.

Post-Training Quantization (PTQ) is the process of compressing those frozen weights. By converting the 16-bit floats into 8-bit or 4-bit integers, you shrink the model's VRAM footprint by 2x to 4x, allowing you to run a 70B model on a single consumer graphics card or a Macbook. Furthermore, because memory bandwidth (moving data from VRAM to the processor) is the primary bottleneck in LLM inference, a 4-bit model generates text significantly faster than a 16-bit model.

Think of It Like This

A high-res photo vs. a compressed JPEG

During training, you are taking a photograph. You want RAW format (16-bit) so you can tweak the exposure, adjust the white balance, and pull details out of the shadows.

But when you text that photo to a friend, you compress it into a JPEG (4-bit quantization). You lose some imperceptible color depth, and if you zoom in 500x you might see artifacts, but it looks exactly the same on their phone and it downloads four times faster.

How It Actually Works

The Bucketing Process

Imagine a layer's weights span from -2.0 to +2.0. In fp16, there are 65,536 possible values in that range. In 4-bit integer (INT4), there are only 16 possible values ($2^4$). Quantization algorithms divide the -2.0 to +2.0 range into 16 "buckets". Every fp16 weight is rounded to the nearest bucket.

Why It Doesn't Destroy the Model

If you round every number in a spreadsheet, the final sum will be wrong. Why does this work for neural networks? Because neural networks are highly over-parameterized and robust to noise. The output of a layer is the sum of thousands of multiplications. The rounding errors (some round up, some round down) tend to cancel each other out during the forward pass.

Advanced PTQ Algorithms (AWQ, GPTQ)

Simple rounding (Round-to-Nearest) degrades performance noticeably at 4-bit. Modern algorithms are smarter:

• AWQ (Activation-Aware Weight Quantization): It looks at a small sample of data passing through the model and identifies the top 1% of weights that are the most "important" (the ones that activate the most). It keeps that 1% in 16-bit precision, and heavily quantizes the rest. This preserves almost all the quality.
• GPTQ: It uses second-order math (the Hessian) to measure how much rounding a specific weight will damage the output, and adjusts the surrounding weights to compensate for the rounding error.

Watch Out For

Activation Quantization

Quantizing the weights saves memory and loads the model faster. But during generation, the activations (the math being done on the user's prompt) are typically converted back to 16-bit in real time to do the multiplication (Weight-Only Quantization). If you quantize the activations as well (e.g., W8A8 - 8-bit weights, 8-bit activations), you get a massive speedup, but it is notoriously difficult to stabilize and often causes severe degradation in LLMs.

The Quick Version

• Post-Training Quantization (PTQ) compresses a fully trained model's weights from 16-bit to 8-bit or 4-bit.
• It reduces the VRAM requirement by 2x to 4x and significantly speeds up text generation.
• Advanced algorithms like AWQ and GPTQ use calibration data to protect the most important weights from being rounded, preserving near-fp16 quality at 4-bit sizes.
• It is the technology that allows powerful open-source models to run locally on consumer hardware.

What to Read Next

• Quantization Quality Tradeoffs explains exactly which capabilities break first when you compress a model too far.