Catastrophic Forgetting in Vision Models
Train a classifier on new classes and accuracy on the old classes collapses toward chance, because the new gradient contains no term for the old loss.
Why Does This Exist?
A product classifier sorts photos into 40 categories and runs well for a year. Then six new categories arrive with a few thousand labeled images each. Retraining from scratch on all 46 costs a full data rebuild every quarter, so you fine-tune the live model on the six new ones. The new classes score beautifully.
Accuracy on the original 40 falls from 0.98 toward chance. Not drift, collapse. The new gradient only mentions the new loss, so weights the old classes depended on move freely, and the shared features that served both get rewritten for the newcomers.
This page names the collapse and measures it. The fixes live in three children: replay methods keep old examples in the batch, regularization methods penalize moving important weights, and architecture methods give tasks separate parameters. The parent overview is continual learning.
Think of It Like This
One whiteboard, two meeting notes
Your team keeps quarterly plans on a shared whiteboard. In January you write the full roadmap across it. In April six new priorities land, so you erase sections and write the new items where they fit. The new plan reads clearly.
The January plan is gone, not faded. Nothing about picking up the eraser consulted it. Replay is photographing the board before erasing and redrawing key lines alongside the new ones. Regularization is writing the old lines in permanent marker so they resist erasing. Separate parameters are buying a second whiteboard. The analogy stops at capacity: whiteboards fill up, while networks overwrite silently even with room to spare.
How It Actually Works
Knowledge of the old classes lives in shared weights, not a labelled compartment. A convolution filter that detects wheel arcs serves old vehicle classes and gets repurposed for the new ones. Move it far and every old decision boundary that read it shifts. Because the new optimum usually sits where old accuracy is near chance, plain gradient descent walks straight there.
Worked example with recomputed numbers
Take a two-stage linear probe over 20 features. Task A reads features 0 to 9, task B reads features 10 to 19. After training A, old-task accuracy is 0.98. Train the same weights on B alone for 600 steps: A falls to 0.50, exactly chance for two classes, because the A-features carried only noise for B and least squares drove them to zero. Now repeat with a replay mix of 24 kept A rows tiled against B: A recovers to 0.71 while B lands at 0.73 instead of 0.96. The 0.23 gap between 0.96 and 0.73 is the stability plasticity price, stated in accuracy points rather than adjectives.
Which fix matches which constraint
When you may keep old images, replay wins and the rest is refinement. When retention policy forbids keeping them, regularization is the honest option. When task identity is known at serving time and parameter growth is affordable, separate parameters remove forgetting by construction. Name the constraint first, then pick.
Watch Out For
Grading only the classes you just trained
The new-class dashboard glows green while old classes fail silently for weeks, because the metric you check after an update structurally cannot see the damage. After every stage, score every earlier task on held out sets and report the full matrix of stages against tasks. Put average accuracy across all tasks so far on the dashboard, since it drops the moment an old task breaks.
Calling slow forgetting a fix
Accuracy sliding from 0.98 to 0.90 over five stages still compounds to failure, and a lower learning rate only stretches the same curve. If old accuracy trends down across stages while new accuracy looks fine, the mechanism is unchanged and the schedule is masking it. Compare against a replay baseline before trusting the schedule.
The Quick Version
- New-task gradients contain no term for the old loss, so shared weights move and old accuracy collapses toward chance.
- The collapse is structural, not noise: one stage can take 0.98 to 0.50.
- Replay mixes old examples into new batches and recovers most of it, here 0.50 back to 0.71.
- Regularization penalizes moving weights the old task needed; separate parameters remove forgetting at the cost of growth.
- Score every old task after every stage, or the failure stays invisible until users find it.