Skip to content
AI360XpertBeta

Catastrophic Forgetting in Fine-Tuning

Fine-tuning on one narrow task rewrites the shared weights behind every other skill, so old abilities fade while the new one climbs. Mixing a little of the old data back into each batch holds the old skills in place.

Fine-tuning on the new task alone drives old scores down, while a 90/10 mix of new and replay data holds them
Fine-tuning on the new task alone drives old scores down, while a 90/10 mix of new and replay data holds them

Why Does This Exist?

Say you have a strong general model and you tune it into an expert on your own task. You feed it thousands of examples in one narrow style, it gets brilliant at that style, and then an old skill quietly breaks. Ask for plain prose and you get your training format pasted where it doesn't belong. Summaries come back wearing the new template like a uniform they cannot take off.

That is catastrophic forgetting: one narrow training distribution rewrites the weights every other skill shared, so the new ability climbs while old ones sink. It bites hardest exactly when fine-tuning works best, because the same gradient steps that carve the new behavior into the model also sand away whatever they pass over.

The one idea you need first: supervised fine-tuning keeps training a pretrained model on demonstration pairs, with the loss computed over the completion tokens. Everything on this page follows from that setup. The loss sees only the current batch, so when every batch comes from one narrow task, the updates serve that task alone.

This page covers why the forgetting happens and how replay data stops it. The training loop itself lives on the supervised fine-tuning page. The most exposed setting, updating every weight at once, is full fine-tuning. The frozen-base alternative is LoRA and QLoRA. The general version of the fix, setting shares for every domain, is data mixing and curriculum.

Think of It Like This

A cover band with one new song

Your band knows forty songs and lands a gig that wants one new hit. For a month you rehearse only that song. Opening night, the new hit sounds incredible. Then someone shouts for an old favorite and the intro falls apart, because a month of rehearsal time went to exactly one place.

The fix is rotation, not longer rehearsals. You spend most of each session on the new hit and keep a couple of old favorites in the set. The old songs stay playable. That kept-in-rotation slot is replay data.

Where it stops: musicians consolidate memory off stage and bounce back fast after a reminder. A model can't do that. It has no second memory, so every update lands in the same shared weights, and every replay example costs real training budget that could have gone to the new task.

How It Actually Works

Shared weights, one loss at a time

Call the trainable weights θ\theta, meaning every number the optimizer may change. On one batch the model suffers an average error LnewL_{\text{new}}, meaning how wrong its predictions are on the new task examples in that batch. Backpropagation returns a gradient g=∇θLnewg = \nabla_{\theta} L_{\text{new}}, meaning one slope per weight saying which way the error climbs. The update steps the other way:

θ←θ−ηg\theta \leftarrow \theta - \eta g

Here η\eta, the step size, says how far each step moves. Nothing in that rule mentions the old skills. When every batch is drawn from the new task, thousands of these steps drag θ\theta wherever the new error is lowest, and the weight patterns that encoded old formats and facts get overwritten along the way. Full fine-tuning is the most exposed version of this, because every entry of θ\theta is free to move.

Replay changes what the gradient protects

Replay mixes old general examples back into each batch. Call wneww_{\text{new}} the fraction of the batch drawn from the new task and wold=1−wneww_{\text{old}} = 1 - w_{\text{new}} the fraction drawn from replay, where both are plain shares between zero and one. The loss becomes a weighted compromise:

L=wnewLnew+woldLoldL = w_{\text{new}} L_{\text{new}} + w_{\text{old}} L_{\text{old}}

Here LoldL_{\text{old}} is the average error on the replayed general examples. Now the gradient carries both errors, so a step that helps the new task but wrecks an old one pays for it immediately. For a batch of BB examples, where BB is the batch size, the expected counts are Enew=BwnewE_{\text{new}} = B w_{\text{new}} and Eold=BwoldE_{\text{old}} = B w_{\text{old}}. With B=100B = 100 and wnew=0.90w_{\text{new}} = 0.90, you get 9090 new examples and 1010 replayed ones in a typical batch. That's the whole trick behind data mixing and curriculum, applied to one new task plus the past.

Freezing weights is the other defense. LoRA and QLoRA leave the base weights untouched and train a small add-on instead, so the old knowledge has nowhere to drain out of. Replay and frozen bases stack well: freeze what you can, replay for the rest.

Worked example: a 30 question toy probe

Take a tiny probe you could recount by hand: 1010 math questions, 1010 code questions, 1010 medical questions. Before tuning, the model gets 88, 88, and 33 right. It is a decent generalist that knows almost no medicine.

Tune it on medical notes only. Afterward it scores 99 on medical, but math slides to 44 and code to 55. The new skill climbed by 66 while the two old ones lost 44 and 33. Those are toy counts, not benchmark claims, and that's the point: you can see the trade directly.

Now repeat the run with a 90/1090/10 mix, meaning 9090 medical examples plus 1010 replayed general ones per 100100. Medical still reaches 99, while math lands at 77 and code at 88. The new task paid almost nothing for the protection. The mixed loss tells the same story with small numbers: if Lnew=0.30L_{\text{new}} = 0.30 and Lold=0.70L_{\text{old}} = 0.70, then

L=0.90×0.30+0.10×0.70=0.27+0.07=0.34L = 0.90 \times 0.30 + 0.10 \times 0.70 = 0.27 + 0.07 = 0.34

The tradeoff is real. Replay dilutes the new-task signal, so training runs a little longer, and someone has to curate replay worth repeating. Skip replay when you actually want the forgetting, such as scrubbing a behavior the model should lose, or when the old skills genuinely don't matter for the deployment. The method assumes the replay pool still represents the skills you care about. When that assumption fails, through stale or too narrow replay, scores on the missing skills keep sinking and the mix number on your config line means nothing.

Code

The point of this snippet is the mixing arithmetic itself: turning a share into per-batch counts and seeing what the compromise loss equals.

def replay_counts(batch_size: int, replay_share: float) -> tuple[int, int]:    replay = int(batch_size * replay_share)    return batch_size - replay, replay

def mixed_loss(new_loss: float, old_loss: float, replay_share: float) -> float:    return (1.0 - replay_share) * new_loss + replay_share * old_loss

new_n, replay_n = replay_counts(100, 0.10)print(new_n, replay_n)                          # -> 90 10print(round(mixed_loss(0.30, 0.70, 0.10), 2))   # -> 0.34

A batch of 100100 at a 0.100.10 replay share carries 9090 new examples and 1010 replayed ones, and losses of 0.300.30 and 0.700.70 blend to 0.340.34. They're the same numbers the worked example computed by hand.

Watch Out For

Replay that never lands in the same batch

You added replay to the dataset and old skills still sink. The usual cause is placement, not share. Replay clustered at the end of training, filtered out by length or packing, or sharded so some workers never see it means many batches train narrow-only. The config says 10 percent. The batches say zero.

Symptom: new-task loss falls smoothly while old-skill evals keep sliding. Fix: interleave replay in every batch and log the sampled fractions from the loader, not the config. If old-skill evals are missing, you won't see forgetting until users do, so keep a small general probe running from the first step.

The format overwrites the facts

Forgetting hits style before facts. Train only on chatty support replies and the model keeps the knowledge but loses the shape: it appends a friendly closer inside your JSON, or answers a plain question in your training template. Parsers break. Evals that check content pass while evals that check shape fail.

Symptom: outputs read right but fail format checks. Fix: make replay format-diverse on purpose, with raw JSON, bare answers, and short replies represented, and gate releases on format checks alongside accuracy.

The Quick Version

  • Catastrophic forgetting means a narrow training distribution rewrites the shared weights behind older skills, so the new ability climbs as old ones sink.
  • The update rule protects only what each batch contains, which is why batches of pure new-task data are the danger zone.
  • Replay data mixes a small share of old general examples into every batch, forcing the gradient to keep both errors low.
  • A 90/10 split of new to replay is a common starting point, with frozen-base methods like LoRA cutting the forgetting further.
  • Replay costs new-task signal and needs curation, and it fails silently when the replay never actually lands in the batch.