Model Quantization for Serving
Weights stored at 16-bit precision can be rounded to 8 or 4 bits. The file shrinks about 4 times, serving gets cheaper and faster, and quality barely moves.
Why Does This Exist?
Weights ship at 16-bit floating point by default: 2 bytes per number. A 7 billion parameter model therefore needs 7 billion times 2 bytes, 14 GB, just to hold its weights. That already strains one good GPU, and a 70 billion model at 140 GB needs several. Most teams do not have several.
Serving is also throttled by memory movement, not arithmetic. Each generated token drags the weights from memory to the compute cores, so smaller weights mean faster tokens as well as cheaper hosting. This page covers the rounding fix: store each weight as an 8-bit or 4-bit integer plus one shared scale, shrinking the file about 4 times with quality nearly intact. The deleting route to small models is model pruning. Training in low precision from the start is mixed precision training. Rounding-aware training that adapts weights to the buckets is quantization-aware training, and the serving-side version of the same idea is post-training quantization.
Think of It Like This
A tailor measuring to the centimetre
A bespoke tailor measures inseams to the millimetre with steel tapes and chalk. A factory tailor measures to the nearest centimetre with a marked stick. The factory suit fits 99 customers in 100, sews far faster, and costs a fraction, because bodies vary more than half a centimetre anyway.
Millimetres are FP16, centimetres are INT8 codes, the stick's printed marks are the shared scale, and the one customer in a hundred with unusual proportions is the math-heavy task that still needs full precision.
Where it stops: the factory still cuts every panel, just measured coarsely. Quantization still stores every weight, just rounded. Neither deletes anything, which is why rounding and pruning stack instead of competing.
How It Actually Works
Call the shared step size the scale, s. With weights clipped to minus 1 through 1, one INT8 code covers a step of s equals 2 divided by 255, about 0.007843. Quantize by dividing and rounding to the nearest whole code; restore by multiplying back. The file stores tiny integers plus one float scale instead of full floats, which is where the 4 times saving comes from.
Three widths cover nearly all serving. FP16 at 2 bytes per weight is the starting point. INT8 at 1 byte halves memory and is nearly free in quality. INT4 at half a byte quarters it and costs a small, measurable dip on hard reasoning. Below that, 2-bit codes collapse and the model stops making sense, so 4-bit is the practical floor.
Worked example: 7 billion weights and one rounded value
Memory first. 7 billion weights times 2 bytes is 14 billion bytes, 14 GB at FP16. Times 1 byte is 7 GB at INT8. Times half a byte is 3.5 GB at INT4. Fourteen to three and a half: one consumer GPU instead of two server cards.
Precision next. Take weight 0.37 with s about 0.007843. Divide: 0.37 divided by 0.007843 is about 47.18, round to code 47. Restore: 47 times 0.007843 is 0.3686. Rounding error is 0.37 minus 0.3686, about 0.0014. That error is the diagram's second half: one number line from minus 1 to 1, the 0.37 mark, and its 0.3686 landing a hair away.
Code
The point is the check: round four real weights to INT8 codes and back, and confirm code 47 returns 0.3686 with the 14 to 3.5 GB shrink.
weights: list[float] = [0.37, -0.82, 0.05, -0.15]scale: float = 2.0 / 255.0
codes: list[int] = [round(x / scale) for x in weights]back: list[float] = [c * scale for c in codes]
print(f"{scale:.6f} {codes}") # -> 0.007843 [47, -105, 6, -19]print([round(x, 4) for x in back]) # -> [0.3686, -0.8235, 0.0471, -0.149]print(f"{7e9 * 2 / 1e9:.1f} {7e9 * 0.5 / 1e9:.1f}") # -> 14.0 3.5The printout matches the hand computation: scale 0.007843, 0.37 becomes code 47 and returns 0.3686, and the model drops from 14.0 to 3.5 GB. Change 0.37 to 0.371 and rerun to watch the code stay 47, which shows how a whole neighborhood of values shares one bucket.
Watch Out For
Pushing past 4-bit and watching reasoning collapse
INT8 is nearly free and INT4 costs a little, so a team tries 3-bit and 2-bit expecting a little more. Instead the model collapses: code and math answers turn to confident nonsense while chitchat still reads fine. Coarse buckets erase exactly the fine distinctions hard tasks need.
Symptom: chat benchmarks barely move at 2-bit while math and code scores fall off a cliff. Fix: treat 4-bit as the floor for anything that reasons, benchmark your own hardest task at each width, and keep 8-bit for legal, medical, or math-heavy serving. The full width-by-width evidence is laid out in precision formats compared, and the inference entry explains why the memory walk dominates serving cost.
Expecting a smaller file to always run faster
A team quantizes to INT4, loads the file on hardware with no INT4 kernels, and watches each token dequantize back to FP16 on the fly. Memory shrinks but every token pays conversion overhead, and latency gets worse than the FP16 original. The shrink is real; the speedup needs matching kernels.
Symptom: 4 times smaller file with equal or slower tokens per second on the serving dashboard. Fix: check the serving runtime for INT8 and INT4 kernel support before picking a width, and measure tokens per second rather than file size. The LoRA-plus-quantization serving recipe that assumes this check is full fine-tuning vs LoRA vs QLoRA.
The Quick Version
- Quantization rounds FP16 weights to INT8 or INT4 codes plus one shared scale, shrinking memory about 4 times.
- Our 7B model drops from 14.0 GB at FP16 to 7 GB at INT8 to 3.5 GB at INT4, fitting one consumer GPU.
- Weight 0.37 divided by scale 0.007843 rounds to code 47 and restores as 0.3686, an error near 0.0014.
- INT8 is nearly free in quality, INT4 costs a little on hard reasoning, and 2-bit collapses, so 4-bit is the practical floor.
- Smaller files speed tokens only on runtimes with matching INT kernels, so measure tokens per second, not gigabytes.