Continual Regularization Methods
When old images cannot be kept, penalize moving the weights the old task needed most, so new learning bends around them.
Why Does This Exist?
Replay needs stored old images, and retention rules often forbid exactly that. Faces, medical frames, and plates carry deletion duties. Regularization methods answer the no-storage case: keep no data, keep only a summary of which weights mattered, and charge the optimizer for moving them.
They repair catastrophic forgetting with memory measured in one importance vector, not a buffer. The sibling that spends parameters instead is architecture methods, under continual learning.
Think of It Like This
Stiff hinges on the old doors
A museum adds a new wing without closing the old halls. Door hinges on the old halls get stiffened in proportion to foot traffic: the grand entrance barely budges, a side closet swings freely. Builders can still reroute corridors, but only through low-traffic joints. The stiffness setting is the penalty strength. The analogy stops at measurement: hinges are tightened by judgment, while weight importance is estimated from the old loss curvature.
How It Actually Works
After the old task finishes, estimate per-weight importance from the curvature of its loss, usually the diagonal Fisher: large curvature means small moves hurt a lot. Then train the new task with the old loss replaced by a quadratic anchor pulling each weight toward its old value with its own strength. Plain weight decay pulls toward zero equally; this pulls toward the old weights unequally.
Worked example with recomputed numbers
Return to the 20-feature probe where plain retraining drops old accuracy to 0.50. Mark features 0 to 9 important with strength 4.0 each and the rest 0.1. With anchor strength set so the penalty matches the new loss scale, old accuracy settles near 0.83 while the new task reaches 0.88 instead of 0.96. Push the strength four times higher and old accuracy climbs to 0.91 while the new task sinks to 0.74. The two runs trace the trade directly: each 0.08 of retained old accuracy costs about 0.14 of new accuracy here.
When the estimate misleads
Fisher importance is local to the old optimum and goes stale as weights travel. Re-estimate per stage on a small held-out old validation split where one exists, and distrust penalties built from tiny or skewed old samples. If importance concentrates on nearly every weight, the tasks genuinely share everything and no penalty setting escapes the trade.
Watch Out For
Setting the strength by new-task accuracy alone
Tuning the penalty to maximize the new task silently drives it toward zero, which is plain fine-tuning with extra code. The symptom is a method that matches the no-protection baseline exactly. Tune on the mean of old and new accuracy, and report both at every setting.
The Quick Version
- Regularization replaces stored data with a per-weight anchor toward old values.
- Importance comes from old-loss curvature: steep directions get held still.
- The worked probe holds old accuracy near 0.83 with the new task at 0.88.
- Strength is the trade dial: more retention costs plasticity, measurably.
- It never beats well-run replay, and it is the honest choice when storage is barred.