Full Fine-Tuning vs LoRA vs QLoRA
Comparing parameter-efficient fine-tuning methods against updating the entire model.
Verdict: Use QLoRA on consumer hardware; use standard LoRA on server GPUs; only attempt Full Fine-Tuning if you have immense compute budgets and are fundamentally pre-training.
The Short Answer
Full Fine-Tuning computes gradients and updates every single weight in a multibillion-parameter LLM, requiring massive clusters of GPUs. LoRA (Low-Rank Adaptation) freezes the original weights and inserts tiny, trainable matrices into the network, cutting memory requirements by 90% while achieving identical quality. QLoRA pushes this further by quantizing (compressing) the frozen base weights down to 4-bit precision, allowing you to fine-tune massive models on a single consumer GPU.
Where They Differ
| Feature | Full Fine-Tuning | LoRA | QLoRA |
|---|---|---|---|
| Weights Updated | All of them (100%) | Only the adapters (~1%) | Only the adapters (~1%) |
| Base Model State | Modified | Frozen (16-bit) | Frozen & Quantized (4-bit) |
| Memory Required | Massive (Multiple A100s) | Moderate | Tiny (Runs on a single RTX 3090) |
| Training Speed | Slow | Fast | Slightly slower than LoRA (due to dequantization) |
Choose LoRA When
- You want to avoid catastrophic forgetting: When you update all weights in a model, it often forgets how to speak normally and overfits wildly to your fine-tuning data. Because LoRA leaves the base weights untouched, the model retains its vast foundational knowledge.
- You are swapping behaviors: You can train multiple LoRA adapters (one for coding, one for summarizing) and swap them in and out of the frozen base model in milliseconds at runtime.
Choose QLoRA When
- You are highly hardware constrained: If you want to fine-tune a 70B parameter model, the optimizer states and gradients alone will instantly crash a standard GPU. QLoRA compresses the base weights to 4-bit NormalFloat, freeing up VRAM to hold the LoRA gradients, making it the king of open-source fine-tuning.
What People Get Wrong
People assume that because LoRA trains fewer parameters, it results in a worse model. Empirically, LoRA achieves parity with Full Fine-Tuning on almost all downstream tasks. The underlying mathematical hypothesis is that the "intrinsic rank" of the changes needed to adapt an LLM to a new task is extremely low, meaning updating all 7 billion parameters was always computationally wasteful.