Gen AI

Full Fine-Tuning

Full fine-tuning unlocks every single parameter in the model and updates them all simultaneously. It is the most powerful way to change a model's behavior, but it requires massive memory and compute overhead.

Why Does This Exist?

When a model's behavior needs to be fundamentally altered—perhaps teaching it a new language, deeply embedding complex medical logic, or executing a massive instruction-tuning phase—you need the maximum possible bandwidth to update its internal representations.

Full fine-tuning (FFT) is the brute-force approach. You load the entire model, unfreeze all of its layers, run forward passes to compute loss, and then backpropagate gradients through the entire network, updating every single weight. It yields the highest possible quality for complex tasks because the model is entirely unconstrained in how it rewires itself.

Think of It Like This

Renovating the entire house vs. painting the walls

Parameter-Efficient Fine-Tuning (PEFT, like LoRA) is like painting the walls and buying new furniture. It changes the look and feel (the behavior and format) of the house quickly and cheaply, but the floor plan remains exactly the same.

Full fine-tuning is a total renovation. You are knocking down walls, moving plumbing, and pouring a new foundation. It takes vastly more resources, but it is the only way to fundamentally change the underlying structure (deep knowledge and complex reasoning patterns) if the original house doesn't fit your needs.

How It Actually Works

The Memory Mathematics

The reason full fine-tuning is rare outside of massive labs is the VRAM (GPU memory) requirement. To train a model, you don't just need memory for the model's weights. You need memory for the gradients and the optimizer states.

If you are training a 7 Billion parameter model in 16-bit precision:

Model Weights: 7B parameters × 2 bytes = 14 GB
Gradients: 7B parameters × 2 bytes = 14 GB
Adam Optimizer States: Adam tracks momentum and variance for every parameter, usually in 32-bit (4 bytes) precision. That's 2 states × 4 bytes = 8 bytes per parameter. 7B × 8 = 56 GB.

So, while a 7B model can run inference on a single 16GB GPU, full fine-tuning it requires over 80GB of VRAM just to hold the math in memory, not even counting the activations from the batch size. This forces engineers to use multi-GPU setups (like FSDP or DeepSpeed Zero) just to tune a relatively small open-weight model.

When is it Actually Justified?

Because it is so expensive, full fine-tuning is only justified when PEFT methods fail. You reach for FFT when:

Continual Pretraining/Mid-training: You are teaching the model an entirely new vocabulary or language.
Deep Reasoning Tasks: The task requires the model to change its fundamental logic pathways, not just its output format (e.g., teaching it advanced mathematics).
Pretraining to SFT: The initial instruction-tuning phase done by the creators of a base model is almost always a full fine-tune to ensure deep behavioral alignment.

Watch Out For

Catastrophic Forgetting

Because you are updating every single weight, full fine-tuning is highly susceptible to destroying the model's original capabilities. If you fully fine-tune a model on 5,000 legal contracts, it might completely forget how to write Python or speak Spanish. PEFT methods are naturally more resistant to this because the original weights remain frozen.

The Quick Version

Full fine-tuning updates every parameter in the neural network.
It provides the highest possible ceiling for learning complex, fundamental tasks.
It requires roughly 4 to 6 times the VRAM of the model's weight size due to gradients and optimizer states.
It is overkill for simple formatting or style adjustments, which should be handled by LoRA.