Skip to content
AI360XpertBeta

PEFT Methods

Freeze the giant model and train only a tiny add-on instead. LoRA, adapters, prefix tuning, and IA3 are four places to put that add-on, and this page maps all four.

PEFT freezes the base model and trains only small injected weights, so a rank-2 LoRA side path on a 32-wide layer trains 128 numbers instead of rewriting 1,024
PEFT freezes the base model and trains only small injected weights, so a rank-2 LoRA side path on a 32-wide layer trains 128 numbers instead of rewriting 1,024

Why Does This Exist?

You want a large model to behave differently, and the two obvious routes both disappoint.

The thorough route is full fine-tuning: unfreeze every weight and retrain all of it. Each weight you move needs its own gradient plus optimizer bookkeeping, so the training bill runs to several times the model itself. The cheap route is prompting: free, instant, and gone the moment the conversation drifts. The model itself never changes.

Parameter-efficient fine-tuning, PEFT for short, is the middle path. You freeze the whole pretrained model, add a tiny set of new trainable numbers somewhere in or around it, and train only those. The knowledge the model already holds stays put. The memory bill shrinks to a fraction. And the change sticks, because it lives in weights rather than in a prompt.

This page maps the four classic places to put that add-on: adapters, prefix tuning, LoRA, and IA3. The technique you'll most likely reach for gets its own deep dive in LoRA and QLoRA. The training loop every method below still runs on lives in supervised fine-tuning.

One grounding note first. Tuning on demonstrations means showing the model input-output pairs and training it to assign high probability to the wanted completion, with the loss counted on the completion tokens. Gradients then say how each trainable number should shift to cut that loss. PEFT changes only which numbers receive those shifts: the add-on, never the base.

Think of It Like This

A rented apartment you are not allowed to renovate

You rent an apartment with great bones and a strict landlord: no tearing down walls. You still need the place to work for night shifts, so you adapt without renovating.

You tape a note by the front door that everyone reads on entry: quiet hours, where the spare key hides. That is prefix tuning, instructions placed where every input passes through. You plumb a small filter into the kitchen tap line, so every drop now flows through your add-on, which also means the flow slows a little. That is an adapter, a bottleneck in series. You clip a compact purifier alongside the tap instead, treating a side stream and blending it back in, and you can unclip it any time. That is LoRA, a parallel path that merges away. Or you leave the hardware alone and adjust the dimmer switches already on the walls. That is IA3, learned rescaling of activations that already exist.

Where it stops: no note, filter, clip-on, or dimmer fixes cracked foundations. If the flat never had hot water, accessories won't install it. Models work the same way. These add-ons steer knowledge the base model already holds. They cannot teach it a subject it never studied.

How It Actually Works

Every method below shares one setup. Call the frozen pretrained numbers WW, meaning every weight and bias you refuse to touch. Call the new trainable numbers θ\theta, pronounced theta, meaning only the add-on. Let PP be the count of all pretrained numbers and SS the count of the add-on numbers. The whole bet of PEFT is S≪PS \ll P: the add-on is orders of magnitude smaller, so its gradients and optimizer states cost almost nothing next to the frozen bulk.

Adapters insert a small bottleneck in series

An adapter is a tiny two-layer network spliced into each transformer block, usually just after the attention sublayer and just after the feed-forward sublayer. Take the block signal hh, a vector of width dd, meaning it holds dd numbers. A down matrix DD with shape mm by dd squeezes it into a narrow bottleneck of width mm, far smaller than dd. A nonlinearity σ\sigma, meaning a simple bend like ReLU applied entry by entry, shapes it. An up matrix UU with shape dd by mm stretches it back, and the result rejoins the signal:

h′=h+U(σ(Dh))h' = h + U(\sigma(Dh))

With a bias on each layer, one adapter holds 2dm+d+m2dm + d + m trainable numbers. Because the signal must pass through it, every token pays one extra hop, which is the price you'll meet again in the pitfalls. The reward is modularity: one frozen base can host many small task adapters you swap at will.

Prefix tuning steers with trainable keys and values

Prefix tuning leaves every weight alone and learns the prompt itself as vectors. Let pp be the count of virtual tokens, meaning learned vectors with no word attached, and dd the model width. You prepend a learned prefix matrix to the keys KK and values VV, meaning the two lists each attention layer compares and blends, in every layer. One injection site then holds p×dp \times d trainable numbers, and the model treats them like context that never changes.

Its lighter cousin prompt tuning prepends those vectors only to the input embeddings. That keeps the architecture untouched but spends real context length on every request. Prefix tuning avoids that particular bill by sitting inside the attention layers, yet it has a reputation for fiddly optimization: small changes to initialization or learning rate can swing a run. When steering from the input side appeals to you, budget extra runs.

LoRA learns a parallel low-rank correction

LoRA freezes a weight matrix WW with shape dd by kk, rows by columns, and bolts on a tiny side path that learns the correction. Call the layer input xx, a vector of length kk. A matrix AA with shape rr by kk squeezes the input down, and a matrix BB with shape dd by rr stretches it back up. The number rr is the rank: a small whole number you choose, far below dd and kk. The adapted output hh is the frozen path plus the side path, with α\alpha, pronounced alpha, a fixed scaling number you choose:

h=Wx+αrBAxh = Wx + \frac{\alpha}{r}BAx

Because the side path runs in parallel, you can fold BABA into WW once training ends, and inference costs nothing extra. That merge is why LoRA became the default. The full treatment, including the 4-bit storage trick, lives in LoRA and QLoRA.

IA3 rescales activations with learned vectors

IA3 is the most frugal of the four: no new matrices at all. It learns one scaling vector ll, meaning one multiplier per activation entry, for the attention keys, one for the values, and one for the feed-forward activations. Each vector multiplies its activations entry by entry, written l⊙hl \odot h, where ⊙\odot means pairwise multiplication of matching entries:

h′=l⊙hh' = l \odot h

Three vectors of width dd hold 3d3d trainable numbers, the smallest add-on on this page. The price is a lower ceiling. Scaling can amplify or mute features that already exist, but it cannot mix them into combinations the base model never computed. Try it cheaply first; reach further when it plateaus.

Worked example: count the trainable numbers

Take one toy weight of width d=32d = 32, a 3232 by 3232 matrix. Updating all of it, the full fine-tuning baseline, trains 32×32=1,02432 \times 32 = 1{,}024 numbers.

Now the four add-ons, each computed directly:

  • LoRA with rank r=2r = 2: AA is 22 by 3232 (6464 numbers) and BB is 3232 by 22 (6464 numbers), for 64+64=12864 + 64 = 128.
  • Adapter with bottleneck m=4m = 4: the down layer holds 32×4=12832 \times 4 = 128 weights plus 44 biases, and the up layer holds 4×32=1284 \times 32 = 128 weights plus 3232 biases, for 128+4+128+32=292128 + 4 + 128 + 32 = 292.
  • Prefix with p=8p = 8 virtual tokens of width 3232: 8×32=2568 \times 32 = 256 per injection site.
  • IA3: three vectors of 3232 multipliers, for 3×32=963 \times 32 = 96.

Every add-on trains less than a third of the 1,0241{,}024, and LoRA trains exactly one eighth. Those are the same counts the diagram and the snippet below use, so all three check each other.

So which add-on should you pick? Default to LoRA: parallel, stable, and free at inference after the merge. Pick adapters when you need many swappable per-task modules and can afford the extra hop. Pick prefix or prompt tuning when the architecture must stay untouched. Pick IA3 for the cheapest first attempt. And pair any of them with instruction tuning: its prompt-plus-answer pairs teach the job while the add-on holds the change.

When should you skip all four? When the model must gain capability it never had. Steering plateaus there no matter how far you raise the rank or lengthen the prefix, and that plateau is the first pitfall below. For deep changes, price full fine-tuning instead.

Code

This snippet recomputes the worked example, so you can check any width, rank, bottleneck, or prefix length of your own before committing to a run.

def full_params(d: int, k: int) -> int:    """Trainable numbers when every entry of a d-by-k weight moves."""    return d * k

def lora_params(d: int, k: int, r: int) -> int:    """A is r-by-k, B is d-by-r; the frozen weight itself trains nothing."""    return r * k + r * d

def adapter_params(d: int, m: int) -> int:    """Down plus up weights, each with its bias vector."""    return 2 * d * m + d + m

def prefix_params(p: int, d: int) -> int:    """p virtual tokens of width d at one injection site."""    return p * d

def ia3_params(d: int) -> int:    """One scaling vector each for keys, values, and feed-forward."""    return 3 * d

d, k, r, m, p = 32, 32, 2, 4, 8print(full_params(d, k))     # -> 1024print(lora_params(d, k, r))  # -> 128print(adapter_params(d, m))  # -> 292print(prefix_params(p, d))   # -> 256print(ia3_params(d))         # -> 96

The five outputs land exactly on the hand counts above: 1,024 for the full matrix against 128, 292, 256, and 96. Change the five inputs to your own layer width and read off which add-on fits your budget.

Watch Out For

Steering a model into knowledge it never had

The symptom is a stubborn plateau. The loss falls, then the behavior stalls, and raising the rank or lengthening the prefix changes nothing. More steering doesn't help because the failure isn't capacity but absence: the base model never learned the subject, so there's nothing to steer.

Stop scaling the add-on and change the setup instead. First check whether the base model can already do the task with a strong prompt; if it can't, no PEFT method will install the missing capability. Teach the base with domain data first, or price full fine-tuning with its forgetting controls. Reserve PEFT for what it does well: style, format, and emphasis on knowledge already inside.

Paying the adapter tax at serving time

The symptom arrives after shipping. A serial adapter adds its hop to every generated token, so tail latency climbs under load, and long prompt-tuned prefixes leave less room for real inputs. Training felt cheap; serving doesn't.

Benchmark the serving path before you commit, not just the training bill. For latency-sensitive routes, prefer LoRA merged back into the weights, where the extra cost is zero. Keep virtual prefixes short, and measure tail latency with the add-on active. The method that trains cheapest isn't always the one that serves cheapest.

The Quick Version

  • PEFT freezes the pretrained model and trains only a small add-on, so tuning costs a fraction of full fine-tuning and the change persists after the prompt ends.
  • Adapters splice a tiny bottleneck into each block in series: modular per task, but every token pays the extra hop.
  • Prefix tuning prepends trainable vectors to attention keys and values, steering without touching weights; prompt tuning does it at the input and spends context length.
  • LoRA trains two small matrices that add a low-rank correction in parallel, then merges them back, so inference costs nothing extra. A rank-2 path on a 32-wide layer trains 128 numbers instead of 1,024.
  • IA3 learns three scaling vectors per block that turn existing activations up or down: the cheapest option, with the lowest ceiling on brand-new behavior.
  • Steering cannot install missing knowledge. When the base model never learned the subject, teach it first or fine-tune everything.