Catastrophic Forgetting in Fine-Tuning
Fine-tuning on one narrow task rewrites the shared weights behind every other skill, so old abilities fade while the new one climbs. Mixing a little of the old data back into each batch holds the old skills in place.
Why Does This Exist?
Say you have a strong general model and you tune it into an expert on your own task. You feed it thousands of examples in one narrow style, it gets brilliant at that style, and then an old skill quietly breaks. Ask for plain prose and you get your training format pasted where it doesn't belong. Summaries come back wearing the new template like a uniform they cannot take off.
That is catastrophic forgetting: one narrow training distribution rewrites the weights every other skill shared, so the new ability climbs while old ones sink. It bites hardest exactly when fine-tuning works best, because the same gradient steps that carve the new behavior into the model also sand away whatever they pass over.
The one idea you need first: supervised fine-tuning keeps training a pretrained model on demonstration pairs, with the loss computed over the completion tokens. Everything on this page follows from that setup. The loss sees only the current batch, so when every batch comes from one narrow task, the updates serve that task alone.
This page covers why the forgetting happens and how replay data stops it. The training loop itself lives on the supervised fine-tuning page. The most exposed setting, updating every weight at once, is full fine-tuning. The frozen-base alternative is LoRA and QLoRA. The general version of the fix, setting shares for every domain, is data mixing and curriculum.
Think of It Like This
A cover band with one new song
Your band knows forty songs and lands a gig that wants one new hit. For a month you rehearse only that song. Opening night, the new hit sounds incredible. Then someone shouts for an old favorite and the intro falls apart, because a month of rehearsal time went to exactly one place.
The fix is rotation, not longer rehearsals. You spend most of each session on the new hit and keep a couple of old favorites in the set. The old songs stay playable. That kept-in-rotation slot is replay data.
Where it stops: musicians consolidate memory off stage and bounce back fast after a reminder. A model can't do that. It has no second memory, so every update lands in the same shared weights, and every replay example costs real training budget that could have gone to the new task.
How It Actually Works
Shared weights, one loss at a time
Call the trainable weights , meaning every number the optimizer may change. On one batch the model suffers an average error , meaning how wrong its predictions are on the new task examples in that batch. Backpropagation returns a gradient , meaning one slope per weight saying which way the error climbs. The update steps the other way:
Here , the step size, says how far each step moves. Nothing in that rule mentions the old skills. When every batch is drawn from the new task, thousands of these steps drag wherever the new error is lowest, and the weight patterns that encoded old formats and facts get overwritten along the way. Full fine-tuning is the most exposed version of this, because every entry of is free to move.
Replay changes what the gradient protects
Replay mixes old general examples back into each batch. Call the fraction of the batch drawn from the new task and the fraction drawn from replay, where both are plain shares between zero and one. The loss becomes a weighted compromise:
Here is the average error on the replayed general examples. Now the gradient carries both errors, so a step that helps the new task but wrecks an old one pays for it immediately. For a batch of examples, where is the batch size, the expected counts are and . With and , you get new examples and replayed ones in a typical batch. That's the whole trick behind data mixing and curriculum, applied to one new task plus the past.
Freezing weights is the other defense. LoRA and QLoRA leave the base weights untouched and train a small add-on instead, so the old knowledge has nowhere to drain out of. Replay and frozen bases stack well: freeze what you can, replay for the rest.
Worked example: a 30 question toy probe
Take a tiny probe you could recount by hand: math questions, code questions, medical questions. Before tuning, the model gets , , and right. It is a decent generalist that knows almost no medicine.
Tune it on medical notes only. Afterward it scores on medical, but math slides to and code to . The new skill climbed by while the two old ones lost and . Those are toy counts, not benchmark claims, and that's the point: you can see the trade directly.
Now repeat the run with a mix, meaning medical examples plus replayed general ones per . Medical still reaches , while math lands at and code at . The new task paid almost nothing for the protection. The mixed loss tells the same story with small numbers: if and , then
The tradeoff is real. Replay dilutes the new-task signal, so training runs a little longer, and someone has to curate replay worth repeating. Skip replay when you actually want the forgetting, such as scrubbing a behavior the model should lose, or when the old skills genuinely don't matter for the deployment. The method assumes the replay pool still represents the skills you care about. When that assumption fails, through stale or too narrow replay, scores on the missing skills keep sinking and the mix number on your config line means nothing.
Code
The point of this snippet is the mixing arithmetic itself: turning a share into per-batch counts and seeing what the compromise loss equals.
def replay_counts(batch_size: int, replay_share: float) -> tuple[int, int]: replay = int(batch_size * replay_share) return batch_size - replay, replay
def mixed_loss(new_loss: float, old_loss: float, replay_share: float) -> float: return (1.0 - replay_share) * new_loss + replay_share * old_loss
new_n, replay_n = replay_counts(100, 0.10)print(new_n, replay_n) # -> 90 10print(round(mixed_loss(0.30, 0.70, 0.10), 2)) # -> 0.34A batch of at a replay share carries new examples and replayed ones, and losses of and blend to . They're the same numbers the worked example computed by hand.
Watch Out For
Replay that never lands in the same batch
You added replay to the dataset and old skills still sink. The usual cause is placement, not share. Replay clustered at the end of training, filtered out by length or packing, or sharded so some workers never see it means many batches train narrow-only. The config says 10 percent. The batches say zero.
Symptom: new-task loss falls smoothly while old-skill evals keep sliding. Fix: interleave replay in every batch and log the sampled fractions from the loader, not the config. If old-skill evals are missing, you won't see forgetting until users do, so keep a small general probe running from the first step.
The format overwrites the facts
Forgetting hits style before facts. Train only on chatty support replies and the model keeps the knowledge but loses the shape: it appends a friendly closer inside your JSON, or answers a plain question in your training template. Parsers break. Evals that check content pass while evals that check shape fail.
Symptom: outputs read right but fail format checks. Fix: make replay format-diverse on purpose, with raw JSON, bare answers, and short replies represented, and gate releases on format checks alongside accuracy.
The Quick Version
- Catastrophic forgetting means a narrow training distribution rewrites the shared weights behind older skills, so the new ability climbs as old ones sink.
- The update rule protects only what each batch contains, which is why batches of pure new-task data are the danger zone.
- Replay data mixes a small share of old general examples into every batch, forcing the gradient to keep both errors low.
- A 90/10 split of new to replay is a common starting point, with frozen-base methods like LoRA cutting the forgetting further.
- Replay costs new-task signal and needs curation, and it fails silently when the replay never actually lands in the batch.