Gen AI

LoRA & QLoRA

Freeze the pretrained model and train only a small pair of low-rank matrices per layer; QLoRA also shrinks the frozen weights to 4-bit so it fits on one GPU.

Freeze the full weight matrix W and train only two small low-rank matrices, A and B, whose product adds a small correction to W's frozen output

Why Does This Exist?

Say you want to turn a 7-billion-parameter open model into a legal-document summarizer, and the only GPU you have is a single consumer card in your desktop. Full fine-tuning means updating every one of those 7 billion parameters, and updating a parameter costs more than storing it: a gradient the same size, plus — if you're using Adam, and almost everyone is — two more running statistics per parameter. Add that up and full fine-tuning needs something like four to six times the raw parameter count in memory, before activations even enter the picture. For a 7B model, that blows past what a consumer GPU carries in VRAM, and you haven't summarized a single contract yet.

That's the actual wall. Renting a bigger GPU works, but it costs money every hour, and it sidesteps a real observation: most of a pretrained model's 7 billion parameters already encode general language ability you don't want to disturb. You're not teaching it English — you're nudging it toward summarizing legal documents in a particular style. LoRA is built on exactly that observation: don't touch the 7 billion parameters at all. Train something much smaller instead, and let that small thing do the adapting.

Think of It Like This

A transparency sheet laid over a printed map

A printed map of a city doesn't get redrawn when your commute changes. Instead, you lay a thin transparency sheet over it and mark your new route in pen — the map underneath stays exactly as printed, and the transparency is the only thing you drew on. Look at the map through the transparency and you see the original streets plus your correction, combined.

That's LoRA's trick. The pretrained model's weights are the printed map: expensive to produce, and left alone. The low-rank matrices are the transparency — small, cheap to write on, stacked on top at read time so the result looks adapted, even though nothing on the map was ever redrawn.

How It Actually Works

Low-rank adaptation, mechanically

Take one weight matrix $W$ inside the model — say, a projection matrix in an attention layer. Full fine-tuning would update $W$ directly: compute a gradient the same shape as $W$ and add it in. LoRA freezes $W$ completely and never touches it again. Instead, it introduces two new matrices, $A$ and $B$ , each far smaller than $W$ , and defines the adapted output as $W$ 's frozen output plus the output of $B$ times $A$ — two small matrices standing in for what would otherwise be a full-sized update.

If $W$ is 4096-by-4096, that's about 16.8 million numbers. A rank-8 LoRA pair — $A$ sized 4096-by-8, $B$ sized 8-by-4096 — comes to roughly 65,000 numbers combined, a few tenths of a percent of $W$ 's size. Only $A$ and $B$ ever get gradients or optimizer state. At inference, the model adds that small correction to $W$ 's output, and the frozen 7 billion parameters do exactly what they always did, with a small nudge layered on top.

Why this works well enough

This only works because of a specific bet: the change a pretrained model needs to specialize toward a new task is far simpler than the model itself. The 7B model already knows grammar, facts, and how to follow instructions — legal summarization doesn't require rewriting any of that, just steering it. That steering lives in a much lower-dimensional space than the full weight matrix, which is exactly what letting $A$ and $B$ stay small is betting on.

It's a real limitation, not a trick you get for free. A low-rank update can't hand the model a capability it fundamentally lacks — it can't teach a model that's never seen a contract what a force majeure clause is if that idea isn't in there somewhere already. What it's good at is redirecting existing capability: tone, format, which details to keep. For the summarizer in the running example, that's exactly the kind of change LoRA suits.

QLoRA's addition

LoRA alone still needs the full 7B frozen model sitting in GPU memory, even though only $A$ and $B$ ever train — and kept in ordinary 16-bit precision, that frozen model is still the majority of the memory bill. QLoRA's fix is to quantize the frozen base model down to 4-bit precision, cutting its footprint by roughly four times, while keeping $A$ and $B$ themselves in a higher precision so ordinary gradient updates still work on them. A paged optimizer handles the memory spikes that show up when a batch briefly needs more than the GPU has free, moving optimizer state to CPU memory instead of crashing.

Put together, this is what makes the running example real: LoRA cuts the trainable count from 7 billion down to a few million, and QLoRA cuts the frozen model's footprint by roughly four times on top. Neither piece alone gets a 7B fine-tune onto one consumer GPU. Both together do.

Show Me the Code

A parameter-count comparison for one weight matrix: the full update versus a rank-8 LoRA pair, for the shape used above.

1def full_update_params(rows: int, cols: int) -> int:2    return rows * cols3
4
5def lora_update_params(rows: int, cols: int, rank: int) -> int:6    # A is (rows x rank), B is (rank x cols) -- only these two ever train7    return rows * rank + rank * cols8
9
10rows, cols, rank = 4096, 4096, 811full = full_update_params(rows, cols)12lora = lora_update_params(rows, cols, rank)13
14print(full)                            # -> 1677721615print(lora)                            # -> 6553616print(round(lora / full * 100, 2))     # -> 0.39

Watch Out For

Picking a rank far higher than the task needs

Cranking rank up to 64 or 128 out of caution increases trainable parameters and cost without a meaningful quality gain, once the true adaptation already fits in a much lower rank. For tone and structure in legal summaries, rank 8 or 16 is often enough — doubling it mostly just doubles the training bill.

Treating a LoRA adapter as portable across base models

An adapter trained against one base model, at one quantization scheme, encodes a correction relative to exactly those frozen weights. Swap the base model out from under it — a newer checkpoint, a different quantization — and the adapter's correction lands on outputs it was never computed against, producing quietly degraded output instead of a clean error.

The Quick Version

LoRA freezes the pretrained model entirely and trains only two small low-rank matrices per targeted weight, added on top at inference.
Trainable parameter count is a tiny fraction of the full matrix's size — rank 8 on a 4096x4096 matrix is under half a percent.
It works because the change needed to adapt a pretrained model is lower-dimensional than the model itself, not because small matrices can hold new capability out of nowhere.
QLoRA quantizes the frozen base model to 4-bit to cut its memory footprint, while still training the LoRA matrices in higher precision.
Paged optimizers absorb memory spikes during training, which combined with 4-bit quantization is what fits a 7B fine-tune onto a single consumer GPU.