LoRA for Image Models: Tiny Adapters
Teach a frozen image model one person, pet, or style with a tiny add-on you can plug in or pull out, instead of retraining the whole model.
Why Does This Exist?
Your base model draws a fine dog. It doesn't draw your dog, with the torn right ear and the gray patch over one eye. Name her in the prompt and you'll get a handsome stranger.
You could rewrite the whole denoiser for her. That means unfreezing every block, paying gradients and optimizer state for every weight, and saving a second full copy that draws her well and everything else a little worse. Repeat for five subjects and you're storing five giants. Full fine-tuning earns that price when the whole distribution must move, not when you want one add-on.
LoRA keeps the base frozen and trains a small side path inside the attention weights. You load it to add one face or style, turn its strength down to soften it, or pull it off to get the base back. This page shows that side path and its counts. The loop it plugs into is diffusion models. Its language model sibling is LoRA and QLoRA. The choice of whether to tune at all is when to fine-tune.
Think of It Like This
The office printer and the overlay sheet
You share an office printer that knows every standard font. It prints clean memos, but it has never seen your handwriting. You don't rebuild the printer to get one birthday card in your script.
Instead you draw your letters once on a transparent sheet. Lay the sheet over any printout and the page reads in your hand. Lift it off and the printer is unchanged. The printer is the frozen diffusion weight, the sheet is the LoRA pair, the thin ink pattern is the low rank, and a slider that fades the sheet is the strength scale.
Where it stops: a real sheet sits on top of pixels. LoRA lives inside the matrix math, before any pixel exists, so a bad sheet can warp faces and backgrounds you never meant to touch. And unlike plastic sheets, two overlays don't always stack cleanly. Styles can fight.
How It Actually Works
Three facts carry it: nothing in the base moves, the change is a thin side path, and a single scale sets how loudly that path speaks.
The frozen weight and the side path
Call the frozen base weight, a table of numbers with rows for outputs and columns for inputs. Call the layer input, a list of numbers. The base output is , the usual matrix times vector.
LoRA never touches . It adds two small tables. Call the down projection, with rows and columns, meaning it squeezes numbers into . Call the up projection, with rows and columns, meaning it stretches those numbers back to . Call the rank, the bottleneck width you pick, like 4 or 8 or 16. Call , pronounced alpha, the strength setting you pick. Call the effective scale, a single number. Call the layer output.
In plain words: run the frozen path, run the thin path, scale the thin result, add. The folded weight is , where is the effective change. Training updates only and . gets no gradients and no optimizer state. Setup matters: starts from small random values and starts at zero, so starts at zero and training begins as the base model.
Worked example: 512 by 512 with rank 8
Take one attention projection you would actually find in an image model. Let outputs and inputs, so holds numbers. Pick and , so .
Then is by and holds numbers. is by and holds numbers. Together that is trainable numbers. That is , about percent, or times fewer than retraining . The diagram draws exactly these counts.
One forward pass on toy numbers, so you see the add. Let , , , , . Then , because and . And , because . So , and . The snippet below recomputes this.
Where the adapter plugs in
Image LoRAs live in attention. In a U-Net or transformer denoiser you'll attach them to the query, key, value, and output projections, in both self attention, which shapes texture and detail, and cross attention, which carries the prompt. That's where a name or style token steers pixels, so that's where a small nudge goes furthest.
Rank sets capacity. Start low, like or for one face or style, and raise it only if the likeness won't land. Alpha sets loudness through . Many setups fix so in training, then expose a sample time weight you can slide. At you have the base. At you have the trained effect. Past you are extrapolating past training, and most image LoRAs overpower the base there.
The trade is plain. You buy cheap, swappable personalization for narrow capacity and fresh tuning knobs. You assume the change you need fits in a thin subspace. When that fails, no rank will save you.
So don't reach for this when the base never learned the structure. A new species from scratch, hands the base mangles everywhere, or a whole domain shift wants broader training or a different base. Don't reach for it for layout either. If the job is pin this pose or this edge map, use ControlNet and spatial conditioning, which pins positions instead of retuning style. If the behavior must live in the weights with nothing loaded at request time, compare full fine-tuning and when to fine-tune.
Code
This snippet recomputes both numbers from the worked example: the parameter saving and the toy forward pass. It uses only NumPy, no model library.
import numpy as np
def lora_params(d: int, k: int, r: int) -> int: return r * (d + k)
print(lora_params(512, 512, 8)) # -> 8192print(512 * 512) # -> 262144
W: np.ndarray = np.array([[1, 0, 2], [0, 1, 0]])A: np.ndarray = np.array([[1, -1, 0]])B: np.ndarray = np.array([[2], [1]])x: np.ndarray = np.array([1, 2, 3])
base = W @ xadapt = B @ (A @ x)print(base) # -> [7 2]print(adapt) # -> [-2 -1]print(base + adapt) # -> [5 1]It prints 8192 and 262144, then , , and , matching the hand counts above. Change to 16 and watch the first number double. That is the capacity knob you are paying for.
Watch Out For
The background sticks to the subject
Symptom: you train on your dog and every sample brings the same couch. The prompt says moon and the image ships a living room. Or a poster style keeps its thick border even when you ask for a clean portrait. The adapter learned the backdrop as part of the identity.
Fix: vary what should vary. Shoot the subject against plain walls, outdoors, and cropped tight. Caption the parts that must change and keep one trigger word for the part that must stay. Hold out two prompts with new places and rerun them every few checkpoints. If the couch survives, cut steps, lower rank, or drop repeats before you blame the base.
Full strength breaks faces and hands
Symptom: at weight 1.0 or higher the likeness looks strong in the preview, then skin turns waxy, fingers bend wrong, and colors oversaturate. Lowering steps after the fact doesn't fix it. The adapter is shouting over the base.
Fix: start below full strength and move in small steps while watching hands and eyes, not just likeness. Do not judge strength on the training prompts alone. Combine two LoRAs at modest weights rather than pushing one past full. Merge into the base only when a single style must ship as one file.
The Quick Version
- LoRA freezes each image model weight and trains only a thin side path, so the base keeps its general skill while the add-on carries one subject or style.
- One 512 by 512 projection holds 262,144 numbers, while a rank 8 pair holds 8,192, about 3.1 percent and 32 times fewer to train.
- Each layer computes , with , and starts as the base model because starts at zero.
- Adapters plug into attention projections, where prompt tokens steer pixels, and a sample time weight below full strength usually beats full strength.
- Small rank assumes the needed change fits in a thin subspace, so new anatomy or a full domain shift needs broader training instead.
- Shared backdrops leak into the trigger and high weights warp faces, so vary training scenes and tune strength on held out prompts.