Skip to content
AI360XpertBeta

Post-Training Quantization

Training needs 16-bit precision for tiny gradient updates. Once weights are frozen, you can round them to 8-bit or 4-bit integers and cut memory severalfold with no retraining.

Post-training quantization rounds each 16-bit weight to the nearest 4-bit level spaced by scale 0.2, so 2.72 becomes code 14 that reads back as 2.8
Post-training quantization rounds each 16-bit weight to the nearest 4-bit level spaced by scale 0.2, so 2.72 becomes code 14 that reads back as 2.8

Why Does This Exist?

A 70 billion parameter model stored at 16 bits per weight needs 140 GB of memory just for the weights: 70 billion times 2 bytes. That is the easy part to picture. The harder part is what happens next: to produce each token, the hardware must move those weights from memory to the compute cores, and it pays that movement cost again for every single token. When the weights are that large, memory movement, not arithmetic, sets your speed. This is the same bottleneck speculative decoding attacks from a different direction.

Training genuinely needs the precision. Gradient updates are tiny, and rounding them away means the model stops learning. But once training ends, the weights freeze. They become a lookup table that inference only reads. Post-training quantization (PTQ) rounds that frozen table to 8-bit or 4-bit integers after training, with no retraining step. Eight bits halves the memory, four bits quarters it, and the per-token movement shrinks by the same factor.

The one idea you need first

A bit width is a budget for distinct values. With bb bits you can name 2b2^b different values: 8 bits gives 256 levels, 4 bits gives 16. A 16-bit float spreads its levels with extra density near zero, while a 4-bit integer code just names one of 16 evenly spaced marks. PTQ is the job of choosing those marks well and rounding each weight to the closest one.

This page covers that rounding mechanism: the scale, the codes, and the worked numbers. What the rounding costs you lives in quantization quality tradeoffs, and the version that pairs 4-bit storage with trainable adapters lives in LoRA and QLoRA.

Think of It Like This

A recipe copied onto coarse measuring cups

While you develop a recipe, you weigh everything on a digital scale that reads to the gram. That is training at 16-bit: you need fine marks because small adjustments change the result.

Now you print the recipe for friends, and their cups only have a mark every 0.2 cups. An amount of 2.72 cups rounds to the nearest mark, 2.8. Each dish shifts a little, and across many ingredients the small overs and unders partly cancel. That printed card is the quantized model: the same recipe, coarser marks, far easier to carry.

Where it stops: your friends all share one set of cups, but PTQ sets the spacing per channel from real calibration text, and methods such as GPTQ and AWQ give extra protection to the few channels that matter most. One global ruler would clip those outliers. The real procedure measures first, then rounds.

How It Actually Works

You pick a range, divide it into evenly spaced levels, and round every weight to the closest level. Two numbers control the whole thing.

The scale sets the ruler, the code picks the mark

Call the bit width bb, so the number of levels is L=2bL = 2^b: 16 levels at 4-bit. Call one original weight xx. From a short run of representative text, called calibration, you read off the smallest and largest values you intend to keep, xmin⁡x_{\min} and xmax⁡x_{\max}. The scale ss is the spacing between neighboring levels, the distance between two adjacent cup marks:

s=xmax⁡−xmin⁡2b−1s = \frac{x_{\max} - x_{\min}}{2^b - 1}

In plain words: the full range divided by the number of gaps between levels. The zero point zz is the code that stands for real zero, which matters when the range crosses zero. When the range starts at zero, zz is simply 0 and the formulas below use xmin⁡x_{\min} directly.

Each weight becomes an integer code xqx_q by shifting, dividing by the scale, rounding, and clipping into range:

xq=clamp(round(x−xmin⁡s),0,2b−1)x_q = \mathrm{clamp}\left(\mathrm{round}\left(\frac{x - x_{\min}}{s}\right), 0, 2^b - 1\right)

Here round\mathrm{round} means nearest integer and clamp\mathrm{clamp} means clip anything outside into the allowed span. At inference time the hardware reads the code back as a reconstructed weight x^\hat{x}:

x^=xmin⁡+s⋅xq\hat{x} = x_{\min} + s \cdot x_q

The rounding error on any single weight, e=x^−xe = \hat{x} - x, can never exceed half a step: ∣e∣≤s/2|e| \le s/2. That bound is the whole game. Everything below is about making ss small where it matters.

Worked example: four weights at 4-bit

Take b=4b = 4, so L=16L = 16 levels, and a channel whose calibration range runs from xmin⁡=0x_{\min} = 0 to xmax⁡=3.0x_{\max} = 3.0. The scale is s=3.0/15=0.2s = 3.0 / 15 = 0.2. Four trained weights come out as 2.722.72, 0.410.41, 1.581.58 and 1.021.02. Quantize each one:

  • 2.72/0.2=13.62.72 / 0.2 = 13.6, which rounds to code 1414. It reads back as 14×0.2=2.814 \times 0.2 = 2.8. Error: 0.080.08.
  • 0.41/0.2=2.050.41 / 0.2 = 2.05, which rounds to code 22. It reads back as 0.40.4. Error: −0.01-0.01.
  • 1.58/0.2=7.91.58 / 0.2 = 7.9, which rounds to code 88. It reads back as 1.61.6. Error: 0.020.02.
  • 1.02/0.2=5.11.02 / 0.2 = 5.1, which rounds to code 55. It reads back as 1.01.0. Error: −0.02-0.02.

The worst error is 0.080.08, inside the bound s/2=0.1s/2 = 0.1. Those four codes are the diagram above: 14, 2, 8, 5.

Now scale it up. Going from 16 bits to 4 bits per weight cuts the storage to one quarter, and a 70 billion parameter model drops from about 140 GB toward about 35 GB. Because decoding reloads the weights for every token, the bytes moved per token fall by the same factor, which is where the speedup comes from.

Why the model survives, and what calibration adds

A single rounding looks harmless, and the reason it stays harmless is arithmetic, not luck. Each error is bounded by half a step and points either way, so when a layer adds up thousands of products, much of the noise averages down instead of piling up. That averaging is also why size matters: a huge layer has far more terms to average over than a tiny classifier, so big models typically tolerate 8-bit rounding with little visible change, while 4-bit needs more care.

Calibration is what makes 4-bit work. Instead of one global range, you set ranges per channel or per small group from representative text, because one outlier channel with huge values would otherwise stretch the scale and drown its neighbors in rounding error. Two families of methods build on that. AWQ first rescales the few channels with the largest activations so their rounding hurts less, then quantizes. GPTQ quantizes one column at a time and pushes each column's rounding error into the columns still to come, using curvature information to size the correction.

The trade is explicit: you accept a small, bounded numeric noise in exchange for roughly a quarter of the memory and movement. Skip PTQ, or stay at 8-bit, when you still need gradients to flow, when the model is so small there is nothing to average over, or when your workload is the kind small flips hurt most: exact code, long chats where errors compound, or weaker languages. Those uneven costs are the subject of quantization quality tradeoffs. And if your goal is fine-tuning rather than serving, keep the 4-bit weights frozen and train small adapters on top instead, as in LoRA and QLoRA.

Code

The mechanism in a few lines: quantize four weights, read them back, and check the worst error against half a step.

import numpy as np

def quantize(x: np.ndarray, s: float, levels: int) -> np.ndarray:    """Round each weight to the nearest level index, then clip into range."""    return np.clip(np.round(x / s), 0, levels - 1).astype(int)

w: np.ndarray = np.array([2.72, 0.41, 1.58, 1.02])s: float = 0.2  # (3.0 - 0.0) / 15, the 4-bit step for range 0 to 3q: np.ndarray = quantize(w, s, 16)x_hat: np.ndarray = q * s  # xmin is 0, so reading back is code times scaleprint(q)  # -> [14  2  8  5]print(x_hat)  # -> [2.8 0.4 1.6 1. ]print(round(float(np.abs(x_hat - w).max()), 2))  # -> 0.08

The codes match the worked example exactly, and 0.08 sits inside the 0.1 bound. Change the range to 0 to 6 and watch the bound double: that is calibration pressure in one number.

Watch Out For

Calibrating on friendly text, then shipping to real prompts

Symptom: perplexity on your calibration sample looks fine, but code answers, long conversations, or another language fall apart after quantization.

The ranges were fit to the wrong distribution. Calibration text that is short, clean English sets tight ranges, while real prompts with outliers land outside them and get clipped, which is a far bigger error than rounding. Evaluate the quantized model under the conditions you will actually ship, not on the calibration set, and recalibrate on a representative mix when the gap shows up. The full checklist for uneven damage is in quantization quality tradeoffs.

Expecting 4-bit arithmetic when you bought 4-bit storage

Symptom: memory drops fourfold but tokens do not speed up as much as you hoped, or an aggressive weight-plus-activation setup degrades quality sharply.

Most LLM PTQ is weight-only: weights rest in 4-bit, then each block is converted back to 16-bit just in time for the math. You save storage and movement, not arithmetic, and that is still a large win because decoding is movement-bound. Quantizing activations too can remove the conversion, but activations carry outliers that make their ranges unstable, so treat activation quantization as a separate decision with its own kernels and checks, not a free extra step.

The Quick Version

  • Post-training quantization rounds frozen weights to 8-bit or 4-bit integers after training, with no retraining step.
  • A scale sets the spacing between levels and a zero point marks where real zero sits. Each weight becomes the nearest code and reads back as code times scale.
  • Any single weight lands within half a step of the truth, and across thousands of products per layer much of that noise averages down.
  • Four bits per weight means roughly one quarter the memory: a 70 billion parameter model moves from about 140 GB toward about 35 GB, and per-token movement shrinks with it.
  • Calibrate ranges on representative text, per channel or group, and confirm quality in ship conditions, since code, long context, and weaker languages feel the noise first.