Gen AI

PEFT Methods

Parameter-Efficient Fine-Tuning (PEFT) is an umbrella term for finding the cheapest possible place to inject trainable parameters into a frozen model. LoRA does it with matrices; other methods do it with prefixes, adapters, or scaling vectors.

Why Does This Exist?

Full fine-tuning is too expensive, and prompt engineering is too transient (it forgets the instructions as soon as the context window clears). We need a way to permanently alter a model's behavior without updating its billions of base parameters.

Parameter-Efficient Fine-Tuning (PEFT) techniques solve this. Instead of unfreezing the massive base model, PEFT methods freeze it entirely. They then introduce a very small number of new, trainable parameters (usually < 1% of the total size) into the architecture. By only training these tiny additions, you can achieve performance comparable to full fine-tuning using a fraction of the VRAM.

While LoRA is the undisputed king of PEFT today, the field produced several distinct architectural approaches to get here.

Think of It Like This

Hacking the pipeline

If the base model is a massive, complex manufacturing assembly line that you are not allowed to rebuild (frozen weights), how do you change the product that comes out?

Prompt Tuning/Prefix Tuning: You put a special supervisor at the very start of the conveyor belt holding a sign that says "Make it blue."
Adapters: You insert a tiny, custom paint-sprayer station between the massive existing machines.
IA3: You turn the volume knobs up or down on the existing machines.
LoRA: You attach a small, parallel conveyor belt that bypasses a main machine, applies a delta, and merges back in.

The PEFT Landscape

1. Adapters (The Original PEFT)

Introduced around 2019, Adapters were the first major PEFT technique. An adapter is a tiny bottleneck feed-forward neural network (two linear layers) inserted directly into each transformer block, usually after the Attention layer and after the FFN layer.

Pros: Highly effective; completely modular.
Cons: Inference latency. Because the adapter is inserted sequentially into the architecture, the forward pass has to stop, run through the adapter, and continue. This adds measurable slowdown during text generation.

2. Prefix Tuning and P-Tuning

Instead of changing the weights, what if we just optimize the prompt? In Prefix Tuning, virtual, trainable tokens are prepended to the input sequence (or the keys/values in the attention mechanism). Instead of human words, these tokens are just continuous vector embeddings optimized by backpropagation to trigger the desired behavior.

Pros: Very low parameter count; leaves the model architecture untouched.
Cons: It consumes part of the context window. If you use 50 virtual tokens, you lose 50 tokens of actual context. It is also notoriously unstable to train compared to weight-updating methods.

3. LoRA (Low-Rank Adaptation)

LoRA bypassed the sequential latency of Adapters. Instead of inserting a layer after the base weights, LoRA attaches two low-rank matrices in parallel to the base weights.

Pros: Zero inference latency (the LoRA matrices can be mathematically merged into the base weights after training). Extremely stable training.
Cons: Merging limits you to one active adapter at a time per base model instance (unless using dynamic un-merging systems like multi-LoRA serving).

4. IA³ (Infused Adapter by Inhibiting and Amplifying Inner Activations)

IA³ is a fascinating, extreme form of PEFT. Instead of adding matrices, IA³ learns a simple vector of scaling factors that multiply against the attention keys, values, and FFN activations. It literally just turns the "volume" up or down on specific neurons.

Pros: Even fewer parameters than LoRA. Extremely fast to train.
Cons: Because it only scales existing features rather than creating new linear combinations, its ceiling for learning entirely novel behaviors is lower than LoRA.

The Quick Version

PEFT freezes the base model and trains a tiny fraction of inserted parameters.
Adapters insert bottleneck layers sequentially (adds inference latency).
Prefix Tuning optimizes continuous virtual tokens at the start of the prompt (consumes context window).
LoRA trains low-rank matrices in parallel and merges them (zero inference latency, current industry standard).
IA³ learns scaling vectors to amplify or inhibit existing activations (lowest parameter count).