Skip to content
AI360XpertBeta

Full Fine-Tuning

You update every weight in the model at once, so it can absorb behavior a small adapter cannot reach. You pay for that reach with the full training memory bill.

Full fine-tuning updates every weight, so a 7B model needs about 84 GB: 14 GB of weights plus 14 GB of gradients plus 56 GB of Adam states
Full fine-tuning updates every weight, so a 7B model needs about 84 GB: 14 GB of weights plus 14 GB of gradients plus 56 GB of Adam states

Why Does This Exist?

You have tried the cheap fixes and the model still won't hold the new behavior. Prompts drift across phrasings, retrieved pages patch facts without changing habits, and a small adapter steers tone without installing capability that was never there. When the change you need lives deep in the network, early layers included, nothing frozen will get you there.

Full fine-tuning is the direct answer: unfreeze every layer, run your data through, and let backpropagation rewrite all of it. You get the highest ceiling of any tuning method because no weight is off limits. You also inherit the highest bill, because every weight you move needs its own gradient plus optimizer bookkeeping, and you're moving all of them.

This page covers that full-update mechanism and its memory arithmetic. The demonstration loop it runs on lives in supervised fine-tuning. The frozen-base alternative lives in LoRA and QLoRA. The failure this invites lives in catastrophic forgetting. The decision of whether the price is worth it lives in when to fine-tune. Working rule: don't reach for this when a sharper prompt, retrieved context, or a small adapter would fix the failure. Reach for it when the model must behave differently on its own, with nothing whispered in at request time, and you can afford the forgetting controls that come with that.

One prerequisite, briefly. Tuning on demonstrations means showing the model input-output pairs and training it to assign high probability to the wanted completion, with the loss counted over the completion tokens only. Gradients then say how each weight should shift to cut that loss. If that loop is hazy, read the supervised fine-tuning page first, then come back.

Think of It Like This

Retraining a whole orchestra, not handing out one new part

A city orchestra has played classical programs for years when the season demands modern jazz. You could hand the two saxophonists a new chart and leave everyone else untouched. That is fast and cheap, and the strings keep bowing exactly as before. But if the brief is a new sound from the entire ensemble, timing and phrasing included, two new charts won't carry it. Every section must rehearse the new style.

Full fine-tuning is the full rehearsal. The orchestra is the pretrained model, each section a layer of weights, the rehearsal schedule your curated data, and the conductor's corrections the gradients telling every player what to adjust. Nothing stays frozen, so the new style can settle everywhere it needs to, including the rhythm section you never thought about.

Where it stops: musicians keep the old repertoire in their hands while they add the new one, and one reminder brings a rusty piece back. Weights have no second memory. Every correction lands in the same shared numbers, so rehearsing only jazz quietly erases the classical program. That erasure is why the forgetting controls later on this page are not optional.

How It Actually Works

Three ideas carry the whole method: nothing is frozen, every moved weight costs memory beyond its own bytes, and the update rule is the same gradient step you already know, applied everywhere at once.

Every weight learns

Call the trainable numbers θ\theta, pronounced theta, meaning every weight and bias the optimizer may change. Let PP be how many of them there are. On one batch the model suffers an average error LL, meaning how wrong its predictions are on those examples. Backpropagation returns a gradient g=∇θLg = \nabla_{\theta} L, meaning one slope per weight saying which way the error climbs. The plain update steps the other way with a step size η\eta, pronounced eta, meaning how far each step moves:

θ←θ−ηg\theta \leftarrow \theta - \eta g

With Adam, the usual optimizer, two running statistics join in. Call mm the momentum average, meaning a smoothed history of past gradients, and vv the variance average, meaning a smoothed history of squared gradients. Bias-corrected versions m^\hat{m} and v^\hat{v} adjust for the early steps, and a tiny constant ϵ\epsilon, pronounced epsilon, keeps the division safe:

θ←θ−ηm^v^+ϵ\theta \leftarrow \theta - \eta \frac{\hat{m}}{\sqrt{\hat{v}} + \epsilon}

In full fine-tuning θ\theta is everything. No layer is frozen and no adapter carries the change, so gg, mm, and vv exist for every one of the PP numbers. That is the entire difference from adapter methods, and the entire source of the bill.

Why one parameter costs 12 bytes

Storing a weight is not the same as training it. Count the bytes per parameter under the standard mixed-precision recipe. Call bwb_w the bytes holding each weight, bgb_g the bytes holding each gradient, and bmb_m and bvb_v the bytes holding the two Adam states. Call MM the total training bytes before activations:

M=P×(bw+bg+bm+bv)M = P \times (b_w + b_g + b_m + b_v)

With weights and gradients in 16-bit precision, bw=2b_w = 2 and bg=2b_g = 2, meaning two bytes each. Adam keeps its two states in 32-bit precision, so bm=4b_m = 4 and bv=4b_v = 4, meaning eight bytes of bookkeeping per parameter. One trainable parameter therefore costs 2+2+8=122 + 2 + 8 = 12 bytes. Activations from your batch size sit on top of MM, and sharding across GPUs splits the sum without shrinking it.

Worked example: eight parameters, then seven billion

Take a toy with P=8P = 8 parameters so every number stays visible. Weights cost 8×2=168 \times 2 = 16 bytes, gradients cost 8×2=168 \times 2 = 16 bytes, and Adam states cost 8×8=648 \times 8 = 64 bytes. The total is 16+16+64=9616 + 16 + 64 = 96 bytes, before a single activation. Twelve bytes per parameter, exactly as the formula says.

Now scale the same arithmetic to P=7,000,000,000P = 7,000,000,000. Weights need 7,000,000,000×2=14,000,000,0007,000,000,000 \times 2 = 14,000,000,000 bytes, about 14 GB. Gradients match at about 14 GB. Adam states need 7,000,000,000×8=56,000,000,0007,000,000,000 \times 8 = 56,000,000,000 bytes, about 56 GB. Add them: 14+14+56=8414 + 14 + 56 = 84, so about 84 GB of VRAM before activations, which is why a model that serves from one card needs several to train.

One update step, concretely. Say two weights stand at θ=[0.50,−0.30]\theta = [0.50, -0.30], their slopes are g=[0.20,−0.40]g = [0.20, -0.40], and the step size is η=0.10\eta = 0.10. The new values are θ←[0.50−0.10×0.20, −0.30−0.10×(−0.40)]=[0.48,−0.26]\theta \leftarrow [0.50 - 0.10 \times 0.20,\ -0.30 - 0.10 \times (-0.40)] = [0.48, -0.26]. Full fine-tuning applies that same subtraction to all seven billion entries, which is why a step size that felt safe for a small adapter can overshoot here.

Code

This snippet is the bill, executable. It prices any parameter count under the same recipe the worked example uses, so you can check the toy total and the 7B total yourself.

def training_bytes(num_params: int) -> dict[str, int]:    weights = num_params * 2  # fp16 weights, bytes    grads = num_params * 2  # fp16 gradients, bytes    adam = num_params * 8  # m + v in fp32, bytes    return {"weights": weights, "grads": grads, "adam": adam, "total": weights + grads + adam}

tiny = training_bytes(8)print(tiny)  # -> {'weights': 16, 'grads': 16, 'adam': 64, 'total': 96}
seven_b = training_bytes(7_000_000_000)print(seven_b["total"])  # -> 84000000000

The toy prints 96 bytes and the 7B count prints 84000000000 bytes, about 84 GB, matching the worked example and the diagram. Activations add more on top at real batch sizes.

Watch Out For

Narrow data quietly erases old skills

Symptom: the new task keeps improving while old abilities slide. The model pastes your new template onto questions that never asked for it, and probes of old skills you once relied on come back worse than the base model. Nothing errors, so teams notice only when users complain.

Fix: hold out a small probe set for the skills you must keep and score it every few checkpoints, not just the new task. Mix a share of old-distribution examples back into training, the replay remedy that catastrophic forgetting details, and stop or rebalance the moment old scores sag. If replay data is unavailable, that is evidence you should be using a frozen-base method instead.

A learning rate tuned for adapters diverges here

Symptom: the loss spikes or collapses within the first steps, sometimes to NaN, even though the same rate trained a LoRA run calmly. Every one of the PP parameters moves at once, so the aggregate step lands much further from the starting point than an adapter step of the same size.

Fix: start near a tenth of the rate you would use for an adapter run, warm up from near zero over the first few percent of steps, and clip gradients while you watch the first checkpoints. If the loss still jumps, lower the rate before touching the data. Do not raise it to push through.

The Quick Version

  • Full fine-tuning trains every weight in the model, so no layer stays frozen and no adapter carries the change.
  • One trainable parameter costs about 12 bytes under mixed precision with Adam: 2 for the weight, 2 for its gradient, 8 for momentum and variance.
  • A 7B model therefore needs about 84 GB of VRAM before activations: 14 GB of weights, 14 GB of gradients, 56 GB of Adam states.
  • Pick full fine-tuning when the target behavior must live in the weights themselves and survive without prompts or retrieval; skip full fine-tuning when a sharper prompt, retrieved context, or LoRA fixes the failure.
  • Narrow full fine-tuning runs erase old skills, so probe the abilities you must keep every few checkpoints and mix replay data back in.