Paper breakdown

GPTQ

Introduced a highly efficient post-training quantization method that can compress 175B parameter models down to 3 or 4 bits per weight with negligible accuracy degradation.

Paper: GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

Authors: Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh · 2022

Read the paper

GPTQ made running massive models locally on consumer hardware a reality by shrinking weight sizes by 4x.

The Problem

As language models grew beyond 100 billion parameters (like OPT-175B or BLOOM-176B), deploying them became prohibitively expensive. A 175B model stored in 16-bit float (FP16) requires 350GB of GPU memory just to load the weights. That requires at least five 80GB A100 GPUs, making it impossible for consumers or researchers to run these models locally.

To shrink the models, researchers used Quantization (reducing the precision of the weights from 16-bit to 8-bit or 4-bit). However, simply rounding all weights to the nearest 4-bit value (Round-to-Nearest, RTN) severely degrades the model's intelligence. More accurate methods required completely retraining or heavily fine-tuning the model (Quantization-Aware Training), which itself required massive compute.

The Idea

The authors sought a Post-Training Quantization (PTQ) method that was both incredibly accurate and fast enough to run on a single GPU in a few hours.

They built upon an older algorithm called OBD/OBS (Optimal Brain Surgeon), which observed that when you introduce an error by quantizing one weight, you can mathematically adjust the remaining unquantized weights to compensate for that error and preserve the original output.

How It Works

GPTQ quantizes the model one layer at a time. It requires a small calibration dataset (e.g., 128 snippets of text) to see how data naturally flows through the layer.

Hessian Matrix: It calculates the inverse Hessian matrix of the layer's activations, which represents how sensitive the layer's output is to changes in each weight.
Sequential Quantization: It quantizes the weights one column at a time.
Error Compensation: When a column is quantized (introducing a rounding error), GPTQ uses the inverse Hessian to calculate exactly how to update the remaining unquantized columns to perfectly counteract that error.
Lazy Batching: Doing this column-by-column is mathematically correct but incredibly slow on GPUs. The authors introduced a brilliant software engineering trick: they grouped columns into blocks and applied the updates lazily, allowing the algorithm to utilize fast GPU matrix multiplications.

Why It Mattered

GPTQ was a revelation for the open-source AI community. It successfully quantized massive 175B parameter models to 3 or 4 bits per weight with almost zero drop in perplexity, and the entire quantization process took less than 4 hours on a single GPU.

Suddenly, a 65B LLaMA model that previously required 130GB of VRAM could run on a single 40GB consumer GPU.

What Came After

GPTQ became an immediate industry standard, spawning massive community efforts (like the popular TheBloke repository) to distribute GPTQ-quantized versions of every open-weight model.

It competes closely with AWQ (Activation-aware Weight Quantization), which achieves similar results by scaling salient weights rather than updating them via the Hessian. Today, for 4-bit weight-only quantization, GPTQ and AWQ are the two dominant standards in the open-weight ecosystem.