Compositional Generalization

Training saw red cubes and blue spheres. The test asks about blue cubes. Models that compose parts answer correctly; models that memorized wholes fail.

Seen parts recombine to 0.83 on unseen blue cubes but novel pairs still fail

Why Does This Exist?

Datasets cover a fraction of possible combinations: every color with every shape in every arrangement is infeasible. Deployed scenes serve the missing combos immediately, and whole-image memorizers answer them with training-set nearest neighbors instead of understanding.

Compositional generalization measures whether models recombine known parts into novel wholes. It splits data so that parts repeat but combinations do not, then scores the unseen cells. The mechanism parents are object-centric learning and disentangled representations: separate parts and factors are what recombine.

Think of It Like This

A keyboard player transposing a song

A musician who memorized finger positions plays one key brilliantly and collapses a semitone up. A musician who knows scales and chords transposes instantly, because notes and patterns were learned separately.

Whole-image models memorize positions. Compositional models learn the scales (shapes, colors, relations) and play new keys. The analogy stops at difficulty: transposition preserves structure exactly, while visual recombination must also handle new occlusions and lighting the training combos never showed.

How It Actually Works

Build a grid over factors (3 shapes by 3 colors gives 9 cells) and hold out whole cells: train on 6, test on 3 unseen combos. Score per-cell accuracy plus the gap between seen and unseen cells. Architectures that help share the same trait: separate encoders or slots per factor, explicit relation heads, and training penalties that keep factor branches independent.

Worked example

A demo grid trains on red cubes, red spheres, and blue spheres (plus three more cells) and tests on unseen blue cubes: 60 test images. The compositional model scores 50 of 60 (0.833); a whole-image baseline scores 28 of 60 (0.467). Seen-cell accuracy is 0.94 for both, so the 0.366 gap isolates recombination, not perception. A second unseen cell with a brand-new shape (torus, never trained) drops both models below 0.40: recombining known parts works, inventing new parts does not.

Watch Out For

Leaking combos through augmentation

Random color jitter or copy-paste augmentation can smuggle held-out combos into training. The symptom is a compositional headline near seen-cell accuracy with no architectural cause. Audit the augmentation pipeline against the holdout cells before claiming systematicity.

The Quick Version

Compositional tests hold out whole factor combos while repeating the parts.
On the demo grid the compositional model scores 0.833 unseen versus 0.467 baseline.
Seen-cell parity at 0.94 proves the gap is recombination, not perception.
Brand-new parts still fail; recombination needs known pieces.
Audit augmentations for holdout leaks before claiming a win.