Quantization Quality Tradeoffs

Why Does This Exist?

A 70 billion parameter model at 16 bits needs about 140 GB just for weights: 70 billion times 2 bytes. At 4 bits it needs about 35 GB: 70 billion times 0.5 bytes. That gap decides whether the model fits on your cards. Because decoding reloads the weights for every token, it also decides how many bytes you move per token. So you quantize. The saving is real and you will want it.

Here is the failure this page exists to prevent. A team ships a coding and multilingual assistant at 4-bit after a quick English check looks fine. Short answers still look fine. Then long support chats drift, code comes back subtly broken, and quality drops hardest once customers switch away from English. Nothing in that first check predicted it, because the check measured the one condition quantization harms least.

This page is about that uneven cost: why identical numeric noise barely touches some tasks and breaks others, and how to test so you catch it before your users do. The rounding mechanism itself lives in post-training quantization. The way to keep 4-bit weights and still adapt the model lives in LoRA and QLoRA.

You need one idea first. A bit width $b$ is a budget for distinct values. With $b$ bits you can name $L = 2^b$ levels, so 4-bit names 16 levels. Quantization picks a range from real calibration text, spaces those levels evenly, and rounds each weight to the closest mark. If you want the full arithmetic, read the mechanism page above first, then come back. You only need one fact here: each weight lands within half a step of the truth, and that step is the same wherever the weight gets used.

Think of It Like This

How It Actually Works

Quantization rounds every weight to the closest of a few evenly spaced marks. That adds a small error to nearly every multiplication you do. The error size is about the same everywhere. Whether it changes the answer depends on how close the decision was. That is the whole story, and the rest of this section puts numbers on it.

The noise is the same everywhere

Call the bit width $b$, so the count of levels is $L = 2^b$. At 4-bit, $L = 16$. From calibration text you read the smallest and largest values you will keep, $x_{\min}$ and $x_{\max}$. In plain words, the range you refuse to clip. The scale $s$ is the spacing between neighboring marks:

Each original weight $x$ becomes an integer code $x_q$, the index of the closest mark, and reads back at inference time as a reconstructed weight $\hat{x}$. The error $e = \hat{x} - x$ obeys $|e| \le s/2$: never more than half a step. That bound is the same for a weight used in English chat, code, or another language. The task does not set the noise. It only decides what the noise can break.

Why close calls flip first

Look at one next token choice. Call the top score $y_1$ and the runner up $y_2$. The decision margin $m = y_1 - y_2$ says how close the call was. Large $m$ means a confident pick. Small $m$ means a coin toss.

Quantization shifts each score by a small amount $\delta$, where $|\delta|$ is at most the weight error times the activation it multiplies. The activation $a$ is the input number that weight multiplies, so with $a = 1.0$ the shift equals the weight error itself. In the worst case the top score drops by $|\delta|$ while the rival rises by $|\delta|$, so the gap can shrink by $2|\delta|$. If $m$ is larger than that swing, the winner cannot change. If $m$ is smaller, it can. Same noise, different margins, different outcomes. That is why code breaks while short factual answers survive.

Worked example with two margins

Take a channel whose calibration range runs $0$ to $3.0$ at $b = 4$. Then $L = 16$ and $s = 3.0 / 15 = 0.2$. In plain words, marks every 0.2. A trained weight $x = 2.72$ gives $2.72 / 0.2 = 13.6$, which rounds to code $x_q = 14$ and reads back as $\hat{x} = 14 \times 0.2 = 2.8$. The error is $e = 2.8 - 2.72 = 0.08$, inside the bound $s/2 = 0.1$. Those are the numbers in the diagram above.

Feed that weight with activation $a = 1.0$, so each affected score can move up to $0.08$. The gap can swing up to $0.08 + 0.08 = 0.16$.

Say the short answer scores the correct token at $y_1 = 3.20$ and its rival at $y_2 = 2.00$. Then $m = 3.20 - 2.00 = 1.20$. Even if the gap loses the full $0.16$, it stays at $1.20 - 0.16 = 1.04$. Still positive. The answer cannot flip.

Say the code choice scores the correct bracket at $y_1 = 2.05$ and the wrong one at $y_2 = 2.00$. Then $m = 2.05 - 2.00 = 0.05$. A $0.16$ swing covers that easily: $0.05 - 0.16 = -0.11$. The wrong token can move ahead. One token changes, and in code one token can flip a comparison or break a call. You felt the same $0.08$ in both cases. Only the second case had a margin thin enough to lose.

Why long context and weaker languages suffer more

Long context follows the same logic across steps instead of inside one step. Each new token attends over what came before, and the model feeds its own output back as the next input. The per step noise does not grow, but the number of close calls does. A short answer risks one flip. A fifty thousand token run risks thousands, and an early wobble can steer everything after it. You will not see that drift in a five prompt check because you have not given it enough chances to happen. That compounding is also why context degradation deserves its own read.

Languages follow the same rule through redundancy. English has the most training text and the most overlapping ways to say the same thing, so many decisions start with wide margins. A lower resource language has thinner coverage, so more decisions sit close together before you quantize. The same $0.08$ shift flips more of them. You did not break that language on purpose. You applied uniform noise to an uneven starting point.

The trade is plain. You buy roughly a quarter of the memory and movement: a 70 billion parameter model moves from about 140 GB toward about 35 GB, since $70 \times 2 = 140$ and $70 \times 0.5 = 35$. You pay uneven quality. Do not pay it when your ship workload is mostly exact code, long chats where errors feed back, or weaker languages, unless you have measured those buckets and the drop is one you can live with. The method assumes errors stay small, bounded, and partly average away across wide layers. That assumption fails on thin margins, on calibration text that mismatches real prompts, and on tiny models with little to average over.

Code

The check in a few lines: rebuild the worked error, then ask which margins survive a worst case swing of twice that error.

1s: float = 0.2  # 3.0 / 15, the 4-bit step for range 0 to 32w_true: float = 2.723q: int = int(round(w_true / s))4w_hat: float = q * s5e: float = round(w_hat - w_true, 2)6
7print(q)  # -> 148print(round(w_hat, 2))  # -> 2.89print(e)  # -> 0.0810print(1.20 > 2 * 0.08)  # -> True11print(0.05 > 2 * 0.08)  # -> False

The codes match the worked example, and the two booleans tell the story: the 1.20 margin survives the 0.16 swing while the 0.05 margin does not. Use this shape on your own eval: group prompts by bucket, compute the drop per bucket, and gate the ship decision on the worst bucket rather than the average.

Watch Out For

The Quick Version

• Quantization adds the same small rounding noise everywhere, bounded by half a step. On range 0 to 3 at 4-bit, that bound is 0.1.
• Whether noise changes an answer depends on the decision margin, the gap between the top two scores for the next token.
• A 0.08 shift leaves a 1.20 margin short answer safe and can flip a 0.05 margin code token, because the gap can swing up to 0.16.
• Long runs and weaker languages break first because they hold more close calls, not because their per weight noise is larger.
• A 70 billion parameter model drops from about 140 GB toward about 35 GB at 4-bit, which is the saving that makes the uneven cost worth weighing.
• Test quantized builds in ship conditions and gate on the worst bucket, never on a short English check alone.

What to Read Next

• Post-Training Quantization
• LoRA & QLoRA
• Context Degradation