LoRA & QLoRA

Why Does This Exist?

You have one consumer GPU and a 7 billion parameter model you want to turn into a legal-document summarizer. Full fine-tuning updates every weight, and each weight drags along a gradient plus two optimizer states, so the training bill runs several times the size of the model itself. On your card, that job doesn't fit.

Here is the observation that saves you. You're not teaching the model English. It already knows grammar, facts, and how to follow instructions. You only need to steer it: this tone, this format, keep these details. LoRA (low-rank adaptation) turns that steering into something tiny you train, while the 7 billion weights stay frozen. QLoRA goes one step further and stores those frozen weights in 4-bit, so the part you never touch stops eating your memory.

The one idea you need first: this is still supervised fine-tuning. You show the model curated input-output pairs and compute gradients. The only change is which weights receive those gradients: not the base model, just your two small matrices. This page covers that adapter mechanism plus the 4-bit trick. The wider family of such tricks lives in PEFT methods, and compression on its own lives in post-training quantization.

Think of It Like This

How It Actually Works

Pick one weight matrix inside the model, say an attention projection. Full fine-tuning would rewrite it. LoRA freezes it and bolts on a tiny side path that learns the correction instead.

The low-rank update

Call the frozen matrix $W$, with shape $d$ by $k$ (rows by columns). Call the layer input $x$, a vector of length $k$. LoRA adds two new matrices: $A$ with shape $r$ by $k$, which squeezes the input down, and $B$ with shape $d$ by $r$, which stretches it back up. The number $r$ is the rank: a small whole number you choose, far below $d$ and $k$. The adapted output $h$ is the frozen path plus the side path:

In plain words: $Wx$ is what the frozen model always computed. $Ax$ squeezes the input into $r$ numbers. $B$ stretches those back into a full-sized correction. The knob $\alpha$ (alpha) sets how strongly the correction counts, and dividing by $r$ keeps that strength stable when you try different ranks. In the worked example below we set $\alpha = r$, so the scale equals $1$ and $h = Wx + BAx$.

Two details matter in practice. $B$ starts at zero, so on day one the correction is exactly zero and the model behaves like the untouched base. And only $A$ and $B$ ever receive gradients or optimizer state. $W$ is read-only from the first step to the last.

Count the trainable numbers. A full update to $W$ needs $d \times k$ values. The LoRA pair needs $d \times r + r \times k$. Take $d = 4096$, $k = 4096$, $r = 8$: the full update is $4096 \times 4096 = 16{,}777{,}216$ numbers, while LoRA needs $4096 \times 8 + 8 \times 4096 = 32{,}768 + 32{,}768 = 65{,}536$. That ratio is $65{,}536 / 16{,}777{,}216 = 0.00390625$, about $0.39\%$. Same layer, same input, under half a percent of the trainable count.

Why so few numbers can steer so much

This rests on one bet: the change a pretrained model needs for a new style or task is far simpler than the model itself. Grammar, facts, and instruction-following are already in $W$. Tone, format, and which details to keep live in a much smaller space, and that is what $A$ and $B$ span.

Name the trade-off plainly. You buy cheap training with limited reach. When the job needs a deep shift, genuinely new knowledge, or behavior the base model never learned, the small correction runs out of room and full fine-tuning (or better data, or a stronger base) is the honest answer. Don't use LoRA to teach what isn't there. Use it to redirect what is.

What QLoRA adds

LoRA alone still parks the whole frozen model in GPU memory at full precision, and for a 7B model that frozen bulk dominates the bill. QLoRA shrinks it: each frozen weight is stored in 4-bit instead of 16-bit, cutting the frozen bytes to roughly one quarter. The adapters $A$ and $B$ stay in higher precision, because gradient updates need fine steps that 4-bit rounding would destroy. A paged optimizer absorbs the brief memory spikes of a busy batch by parking excess optimizer state in CPU memory instead of crashing.

Neither half works alone. LoRA collapses the trainable count; QLoRA collapses the frozen footprint. Together they put a 7B adaptation on one consumer card.

Worked example

Keep the shapes tiny so you can check every step by hand. Let $W$ be the $2$ by $2$ identity, $x = [2, 3]$, $A = [[2, -1]]$ ($1$ by $2$, so $r = 1$), $B = [[1], [2]]$ ($2$ by $1$), and $\alpha = r = 1$:

Frozen path: $Wx = [1 \times 2 + 0 \times 3,\ 0 \times 2 + 1 \times 3] = [2, 3]$.

Squeeze: $Ax = 2 \times 2 + (-1) \times 3 = 4 - 3 = 1$. One number, as $r = 1$ promises.

Stretch: $B(Ax) = [1 \times 1,\ 2 \times 1] = [1, 2]$.

Add: $h = [2 + 1,\ 3 + 2] = [3, 5]$.

The diagram above draws exactly this: frozen $[2, 3]$ plus correction $[1, 2]$ gives $[3, 5]$. At realistic scale the same addition holds, except the correction comes from $65{,}536$ trained values sitting beside $16{,}777{,}216$ frozen ones.

Code

The snippet below runs the worked example and the realistic parameter count side by side, so you can confirm both numbers yourself:

1def matvec(m: list[list[int]], v: list[int]) -> list[int]:2    return [sum(row[j] * v[j] for j in range(len(v))) for row in m]3
4
5def lora_forward(6    W: list[list[int]],7    A: list[list[int]],8    B: list[list[int]],9    x: list[int],10) -> list[int]:11    base = matvec(W, x)  # frozen path, never trains12    low = matvec(A, x)  # squeeze to rank r13    up = matvec(B, low)  # stretch back to full size14    return [b + u for b, u in zip(base, up)]15
16
17W = [[1, 0], [0, 1]]18x = [2, 3]19A = [[2, -1]]20B = [[1], [2]]21
22print(lora_forward(W, A, B, x))  # -> [3, 5]23print(4096 * 4096)  # -> 1677721624print(4096 * 8 + 8 * 4096)  # -> 65536

The forward pass prints $[3, 5]$, matching the hand computation. The counts print $16{,}777{,}216$ versus $65{,}536$, matching the $0.39\%$ ratio in the prose.

Watch Out For

The Quick Version

• LoRA freezes every pretrained weight and trains only two small matrices, $A$ and $B$, whose product is added to the frozen output.
• With rank 8 on a 4096 by 4096 weight, LoRA trains 65,536 values instead of 16,777,216, which is about 0.39 percent.
• The trick works because steering a pretrained model needs far fewer dimensions than pretraining it, not because small matrices can invent new knowledge.
• QLoRA stores the frozen weights in 4-bit while keeping the adapters in higher precision, cutting frozen bytes to roughly one quarter.
• A paged optimizer parks brief memory spikes in CPU memory, and that plus 4-bit storage is what fits 7B adaptation on a single consumer GPU.
• An adapter belongs to one exact base checkpoint and quantization; a swap means a retrain.

What to Read Next

• Full Fine-Tuning
• Supervised Fine-Tuning
• PEFT Methods
• Post-Training Quantization