Architecture Methods for Continual Vision
Give each task its own parameters, adapters, or masks, so new learning has somewhere to go that is not on top of old knowledge.
Why Does This Exist?
When forgetting is unacceptable and storage is barred, the remaining move is structural: stop sharing everything. Freeze the parameters the old task uses, attach fresh capacity for the new one, and route each input to its own path. Forgetting drops to near zero by construction, because the old path never changes.
The price is growth and routing. Parameters accumulate with every task, and serving must know which task an input belongs to. Compare with replay, which shares everything and stores data, and regularization, which shares everything and stores importance. Parent: continual learning.
Think of It Like This
Adding a new wing instead of rearranging rooms
A library outgrows its shelves. Rearranging the old stacks disturbs every reader, so the library builds a new wing for the new collection and leaves the old halls untouched. Readers need a sign at the entrance pointing to the right wing. The analogy stops at sharing: real adapters reuse the frozen backbone features, while a new wing shares nothing with the old building.
How It Actually Works
Three patterns cover nearly every proposal. Adapters insert small trainable modules per task while the backbone stays frozen. Masks learn a binary routing per task over a shared network, so each task owns a subnetwork. Progressive columns add a whole new column per task with lateral links reading frozen old features. All three freeze old knowledge and learn in fresh capacity; they differ in how much grows and what the router needs.
Worked example with recomputed numbers
Backbone with 1.00M parameters plus one 0.08M adapter per task, so 8 percent growth each. Old task frozen at 0.98 slips only to 0.97 from shared normalization drift, while the new task trains its adapter to 0.95. After five tasks the model holds 1.40M parameters and every old task still scores within a point of its original. The residual cost is exact: 0.40M parameters and a task router at serving time.
Routing is the quiet requirement
At inference the input must reach its task path. With explicit task labels this is trivial. Without them, infer the task from the input first, and that classifier becomes the new single point of failure: route an old image to the new adapter and accuracy collapses despite zero forgetting in the weights.
Watch Out For
Forgetting to budget the router
Teams count adapter parameters and declare victory, then route by argmax confidence and watch old accuracy crater on ambiguous inputs. The symptom is per-task accuracy that is perfect with oracle routing and poor in production. Budget router accuracy as a first-class metric and fall back to a shared head when confidence is low.
The Quick Version
- New capacity per task removes forgetting by construction: old weights never move.
- The worked setup holds old accuracy at 0.97 with the new task at 0.95.
- Cost is explicit growth, here 8 percent per task, plus a router.
- Adapters, masks, and progressive columns differ in growth versus sharing.
- Without reliable task identity at serving time, prefer replay or regularization.