Skip to content
AI360XpertBeta

LoRA for Image Models: Tiny Adapters

Teach a frozen image model one person, pet, or style with a tiny add-on you can plug in or pull out, instead of retraining the whole model.

LoRA freezes a 512 by 512 image weight and trains a rank 8 side path with 8,192 parameters instead of 262,144, adding its output back with scale alpha over r
LoRA freezes a 512 by 512 image weight and trains a rank 8 side path with 8,192 parameters instead of 262,144, adding its output back with scale alpha over r

Why Does This Exist?

Your base model draws a fine dog. It doesn't draw your dog, with the torn right ear and the gray patch over one eye. Name her in the prompt and you'll get a handsome stranger.

You could rewrite the whole denoiser for her. That means unfreezing every block, paying gradients and optimizer state for every weight, and saving a second full copy that draws her well and everything else a little worse. Repeat for five subjects and you're storing five giants. Full fine-tuning earns that price when the whole distribution must move, not when you want one add-on.

LoRA keeps the base frozen and trains a small side path inside the attention weights. You load it to add one face or style, turn its strength down to soften it, or pull it off to get the base back. This page shows that side path and its counts. The loop it plugs into is diffusion models. Its language model sibling is LoRA and QLoRA. The choice of whether to tune at all is when to fine-tune.

Think of It Like This

The office printer and the overlay sheet

You share an office printer that knows every standard font. It prints clean memos, but it has never seen your handwriting. You don't rebuild the printer to get one birthday card in your script.

Instead you draw your letters once on a transparent sheet. Lay the sheet over any printout and the page reads in your hand. Lift it off and the printer is unchanged. The printer is the frozen diffusion weight, the sheet is the LoRA pair, the thin ink pattern is the low rank, and a slider that fades the sheet is the strength scale.

Where it stops: a real sheet sits on top of pixels. LoRA lives inside the matrix math, before any pixel exists, so a bad sheet can warp faces and backgrounds you never meant to touch. And unlike plastic sheets, two overlays don't always stack cleanly. Styles can fight.

How It Actually Works

Three facts carry it: nothing in the base moves, the change is a thin side path, and a single scale sets how loudly that path speaks.

The frozen weight and the side path

Call WW the frozen base weight, a table of numbers with dd rows for outputs and kk columns for inputs. Call xx the layer input, a list of kk numbers. The base output is WxW x, the usual matrix times vector.

LoRA never touches WW. It adds two small tables. Call AA the down projection, with rr rows and kk columns, meaning it squeezes kk numbers into rr. Call BB the up projection, with dd rows and rr columns, meaning it stretches those rr numbers back to dd. Call rr the rank, the bottleneck width you pick, like 4 or 8 or 16. Call α\alpha, pronounced alpha, the strength setting you pick. Call s=α/rs = \alpha / r the effective scale, a single number. Call hh the layer output.

h=Wx+sB(Ax)h = W x + s B (A x)

In plain words: run the frozen path, run the thin path, scale the thin result, add. The folded weight is W′=W+sBAW' = W + s B A, where ΔW=sBA\Delta W = s B A is the effective change. Training updates only AA and BB. WW gets no gradients and no optimizer state. Setup matters: AA starts from small random values and BB starts at zero, so ΔW\Delta W starts at zero and training begins as the base model.

Worked example: 512 by 512 with rank 8

Take one attention projection you would actually find in an image model. Let d=512d = 512 outputs and k=512k = 512 inputs, so WW holds 512×512=262,144512 \times 512 = 262{,}144 numbers. Pick r=8r = 8 and α=8\alpha = 8, so s=8/8=1.0s = 8 / 8 = 1.0.

Then AA is 88 by 512512 and holds 8×512=4,0968 \times 512 = 4{,}096 numbers. BB is 512512 by 88 and holds 512×8=4,096512 \times 8 = 4{,}096 numbers. Together that is 4,096+4,096=8,1924{,}096 + 4{,}096 = 8{,}192 trainable numbers. That is 8,192/262,144=0.031258{,}192 / 262{,}144 = 0.03125, about 3.13.1 percent, or 3232 times fewer than retraining WW. The diagram draws exactly these counts.

One forward pass on toy numbers, so you see the add. Let W=[[1,0,2],[0,1,0]]W = [[1, 0, 2], [0, 1, 0]], A=[[1,−1,0]]A = [[1, -1, 0]], B=[[2],[1]]B = [[2], [1]], x=[1,2,3]x = [1, 2, 3], s=1.0s = 1.0. Then Wx=[7,2]W x = [7, 2], because 1×1+0×2+2×3=71 \times 1 + 0 \times 2 + 2 \times 3 = 7 and 0×1+1×2+0×3=20 \times 1 + 1 \times 2 + 0 \times 3 = 2. And Ax=−1A x = -1, because 1×1+(−1)×2+0×3=−11 \times 1 + (-1) \times 2 + 0 \times 3 = -1. So B(Ax)=[−2,−1]B (A x) = [-2, -1], and h=[7,2]+[−2,−1]=[5,1]h = [7, 2] + [-2, -1] = [5, 1]. The snippet below recomputes this.

Where the adapter plugs in

Image LoRAs live in attention. In a U-Net or transformer denoiser you'll attach them to the query, key, value, and output projections, in both self attention, which shapes texture and detail, and cross attention, which carries the prompt. That's where a name or style token steers pixels, so that's where a small nudge goes furthest.

Rank sets capacity. Start low, like 44 or 88 for one face or style, and raise it only if the likeness won't land. Alpha sets loudness through ss. Many setups fix α=r\alpha = r so s=1.0s = 1.0 in training, then expose a sample time weight you can slide. At 00 you have the base. At 1.01.0 you have the trained effect. Past 1.01.0 you are extrapolating past training, and most image LoRAs overpower the base there.

The trade is plain. You buy cheap, swappable personalization for narrow capacity and fresh tuning knobs. You assume the change you need fits in a thin subspace. When that fails, no rank will save you.

So don't reach for this when the base never learned the structure. A new species from scratch, hands the base mangles everywhere, or a whole domain shift wants broader training or a different base. Don't reach for it for layout either. If the job is pin this pose or this edge map, use ControlNet and spatial conditioning, which pins positions instead of retuning style. If the behavior must live in the weights with nothing loaded at request time, compare full fine-tuning and when to fine-tune.

Code

This snippet recomputes both numbers from the worked example: the parameter saving and the toy forward pass. It uses only NumPy, no model library.

import numpy as np

def lora_params(d: int, k: int, r: int) -> int:    return r * (d + k)

print(lora_params(512, 512, 8))  # -> 8192print(512 * 512)  # -> 262144
W: np.ndarray = np.array([[1, 0, 2], [0, 1, 0]])A: np.ndarray = np.array([[1, -1, 0]])B: np.ndarray = np.array([[2], [1]])x: np.ndarray = np.array([1, 2, 3])
base = W @ xadapt = B @ (A @ x)print(base)  # -> [7 2]print(adapt)  # -> [-2 -1]print(base + adapt)  # -> [5 1]

It prints 8192 and 262144, then [7,2][7, 2], [−2,−1][-2, -1], and [5,1][5, 1], matching the hand counts above. Change rr to 16 and watch the first number double. That is the capacity knob you are paying for.

Watch Out For

The background sticks to the subject

Symptom: you train on your dog and every sample brings the same couch. The prompt says moon and the image ships a living room. Or a poster style keeps its thick border even when you ask for a clean portrait. The adapter learned the backdrop as part of the identity.

Fix: vary what should vary. Shoot the subject against plain walls, outdoors, and cropped tight. Caption the parts that must change and keep one trigger word for the part that must stay. Hold out two prompts with new places and rerun them every few checkpoints. If the couch survives, cut steps, lower rank, or drop repeats before you blame the base.

Full strength breaks faces and hands

Symptom: at weight 1.0 or higher the likeness looks strong in the preview, then skin turns waxy, fingers bend wrong, and colors oversaturate. Lowering steps after the fact doesn't fix it. The adapter is shouting over the base.

Fix: start below full strength and move in small steps while watching hands and eyes, not just likeness. Do not judge strength on the training prompts alone. Combine two LoRAs at modest weights rather than pushing one past full. Merge into the base only when a single style must ship as one file.

The Quick Version

  • LoRA freezes each image model weight and trains only a thin side path, so the base keeps its general skill while the add-on carries one subject or style.
  • One 512 by 512 projection holds 262,144 numbers, while a rank 8 pair holds 8,192, about 3.1 percent and 32 times fewer to train.
  • Each layer computes h=Wx+sB(Ax)h = W x + s B (A x), with s=α/rs = \alpha / r, and starts as the base model because BB starts at zero.
  • Adapters plug into attention projections, where prompt tokens steer pixels, and a sample time weight below full strength usually beats full strength.
  • Small rank assumes the needed change fits in a thin subspace, so new anatomy or a full domain shift needs broader training instead.
  • Shared backdrops leak into the trigger and high weights warp faces, so vary training scenes and tune strength on held out prompts.