Full Fine-Tuning
You update every weight in the model at once, so it can absorb behavior a small adapter cannot reach. You pay for that reach with the full training memory bill.
Why Does This Exist?
You have tried the cheap fixes and the model still won't hold the new behavior. Prompts drift across phrasings, retrieved pages patch facts without changing habits, and a small adapter steers tone without installing capability that was never there. When the change you need lives deep in the network, early layers included, nothing frozen will get you there.
Full fine-tuning is the direct answer: unfreeze every layer, run your data through, and let backpropagation rewrite all of it. You get the highest ceiling of any tuning method because no weight is off limits. You also inherit the highest bill, because every weight you move needs its own gradient plus optimizer bookkeeping, and you're moving all of them.
This page covers that full-update mechanism and its memory arithmetic. The demonstration loop it runs on lives in supervised fine-tuning. The frozen-base alternative lives in LoRA and QLoRA. The failure this invites lives in catastrophic forgetting. The decision of whether the price is worth it lives in when to fine-tune. Working rule: don't reach for this when a sharper prompt, retrieved context, or a small adapter would fix the failure. Reach for it when the model must behave differently on its own, with nothing whispered in at request time, and you can afford the forgetting controls that come with that.
One prerequisite, briefly. Tuning on demonstrations means showing the model input-output pairs and training it to assign high probability to the wanted completion, with the loss counted over the completion tokens only. Gradients then say how each weight should shift to cut that loss. If that loop is hazy, read the supervised fine-tuning page first, then come back.
Think of It Like This
Retraining a whole orchestra, not handing out one new part
A city orchestra has played classical programs for years when the season demands modern jazz. You could hand the two saxophonists a new chart and leave everyone else untouched. That is fast and cheap, and the strings keep bowing exactly as before. But if the brief is a new sound from the entire ensemble, timing and phrasing included, two new charts won't carry it. Every section must rehearse the new style.
Full fine-tuning is the full rehearsal. The orchestra is the pretrained model, each section a layer of weights, the rehearsal schedule your curated data, and the conductor's corrections the gradients telling every player what to adjust. Nothing stays frozen, so the new style can settle everywhere it needs to, including the rhythm section you never thought about.
Where it stops: musicians keep the old repertoire in their hands while they add the new one, and one reminder brings a rusty piece back. Weights have no second memory. Every correction lands in the same shared numbers, so rehearsing only jazz quietly erases the classical program. That erasure is why the forgetting controls later on this page are not optional.
How It Actually Works
Three ideas carry the whole method: nothing is frozen, every moved weight costs memory beyond its own bytes, and the update rule is the same gradient step you already know, applied everywhere at once.
Every weight learns
Call the trainable numbers , pronounced theta, meaning every weight and bias the optimizer may change. Let be how many of them there are. On one batch the model suffers an average error , meaning how wrong its predictions are on those examples. Backpropagation returns a gradient , meaning one slope per weight saying which way the error climbs. The plain update steps the other way with a step size , pronounced eta, meaning how far each step moves:
With Adam, the usual optimizer, two running statistics join in. Call the momentum average, meaning a smoothed history of past gradients, and the variance average, meaning a smoothed history of squared gradients. Bias-corrected versions and adjust for the early steps, and a tiny constant , pronounced epsilon, keeps the division safe:
In full fine-tuning is everything. No layer is frozen and no adapter carries the change, so , , and exist for every one of the numbers. That is the entire difference from adapter methods, and the entire source of the bill.
Why one parameter costs 12 bytes
Storing a weight is not the same as training it. Count the bytes per parameter under the standard mixed-precision recipe. Call the bytes holding each weight, the bytes holding each gradient, and and the bytes holding the two Adam states. Call the total training bytes before activations:
With weights and gradients in 16-bit precision, and , meaning two bytes each. Adam keeps its two states in 32-bit precision, so and , meaning eight bytes of bookkeeping per parameter. One trainable parameter therefore costs bytes. Activations from your batch size sit on top of , and sharding across GPUs splits the sum without shrinking it.
Worked example: eight parameters, then seven billion
Take a toy with parameters so every number stays visible. Weights cost bytes, gradients cost bytes, and Adam states cost bytes. The total is bytes, before a single activation. Twelve bytes per parameter, exactly as the formula says.
Now scale the same arithmetic to . Weights need bytes, about 14 GB. Gradients match at about 14 GB. Adam states need bytes, about 56 GB. Add them: , so about 84 GB of VRAM before activations, which is why a model that serves from one card needs several to train.
One update step, concretely. Say two weights stand at , their slopes are , and the step size is . The new values are . Full fine-tuning applies that same subtraction to all seven billion entries, which is why a step size that felt safe for a small adapter can overshoot here.
Code
This snippet is the bill, executable. It prices any parameter count under the same recipe the worked example uses, so you can check the toy total and the 7B total yourself.
def training_bytes(num_params: int) -> dict[str, int]: weights = num_params * 2 # fp16 weights, bytes grads = num_params * 2 # fp16 gradients, bytes adam = num_params * 8 # m + v in fp32, bytes return {"weights": weights, "grads": grads, "adam": adam, "total": weights + grads + adam}
tiny = training_bytes(8)print(tiny) # -> {'weights': 16, 'grads': 16, 'adam': 64, 'total': 96}
seven_b = training_bytes(7_000_000_000)print(seven_b["total"]) # -> 84000000000The toy prints 96 bytes and the 7B count prints 84000000000 bytes, about 84 GB, matching the worked example and the diagram. Activations add more on top at real batch sizes.
Watch Out For
Narrow data quietly erases old skills
Symptom: the new task keeps improving while old abilities slide. The model pastes your new template onto questions that never asked for it, and probes of old skills you once relied on come back worse than the base model. Nothing errors, so teams notice only when users complain.
Fix: hold out a small probe set for the skills you must keep and score it every few checkpoints, not just the new task. Mix a share of old-distribution examples back into training, the replay remedy that catastrophic forgetting details, and stop or rebalance the moment old scores sag. If replay data is unavailable, that is evidence you should be using a frozen-base method instead.
A learning rate tuned for adapters diverges here
Symptom: the loss spikes or collapses within the first steps, sometimes to NaN, even though the same rate trained a LoRA run calmly. Every one of the parameters moves at once, so the aggregate step lands much further from the starting point than an adapter step of the same size.
Fix: start near a tenth of the rate you would use for an adapter run, warm up from near zero over the first few percent of steps, and clip gradients while you watch the first checkpoints. If the loss still jumps, lower the rate before touching the data. Do not raise it to push through.
The Quick Version
- Full fine-tuning trains every weight in the model, so no layer stays frozen and no adapter carries the change.
- One trainable parameter costs about 12 bytes under mixed precision with Adam: 2 for the weight, 2 for its gradient, 8 for momentum and variance.
- A 7B model therefore needs about 84 GB of VRAM before activations: 14 GB of weights, 14 GB of gradients, 56 GB of Adam states.
- Pick full fine-tuning when the target behavior must live in the weights themselves and survive without prompts or retrieval; skip full fine-tuning when a sharper prompt, retrieved context, or LoRA fixes the failure.
- Narrow full fine-tuning runs erase old skills, so probe the abilities you must keep every few checkpoints and mix replay data back in.