Causal and Structured Vision
Plain classifiers learn backgrounds along with objects. Structured vision separates causes from context, so swapping the background no longer breaks the decision.
Why Does This Exist?
A cow classifier trained on pasture photos learns grass as much as cows, then calls a cow on a beach a boat. The error is not a blurry image or a hard pose: the model learned the wrong cause, because background predicted the label in training.
Causal and structured vision is the map for fixes that separate what causes the label from what merely rode along. Causal representation learning isolates interventions. Disentangled representations split factors into controllable knobs. Object-centric learning splits scenes into objects. Scene graphs name the relations. Composition and symbols ride on top.
Think of It Like This
A recipe versus a restaurant review
A restaurant review says the meal was great near the harbor at sunset. A recipe lists ingredients and steps that work in any kitchen. Reviews predict enjoyment from context; recipes cause it.
Plain classifiers learn reviews: cows come with grass. Structured models learn recipes: the animal's shape causes the label under any sky. The analogy stops at cost: recipes need controlled kitchens (interventions, slot architectures), while reviews are cheap to collect.
How It Actually Works
Residual only: the split and what each branch assumes. Causal methods train or test under interventions that change context while holding the object fixed (recolored backgrounds, swapped scenes). Disentangling methods force latent factors apart so one knob moves pose but not identity. Object-centric methods bind pixels to slots before classifying. Graph methods score subject-predicate-object triples explicitly. Each branch prices the same trade: intervention data or architectural cost in exchange for context robustness.
The numbers that order them
On a demo set of 200 animals pasted on swapped backgrounds: a plain classifier falls from 0.93 matched to 0.61 swapped (a 32-point context penalty). A structured pipeline (segment, then classify the masked object) holds 0.88 swapped at twice the inference cost. So the page's rule: report matched and swapped accuracy side by side, with the inference price, or the robustness claim is incomplete.
Watch Out For
Claiming causality from a swapped test set alone
A swapped evaluation measures robustness, not causal discovery. The symptom is a causal headline over a masking trick. Reserve causal claims for models with stated interventions or graphs; call the rest context-robust.
The Quick Version
- Plain classifiers learn context as cause and break when context swaps.
- Structured vision splits objects, factors, and relations before deciding.
- On the demo set plain accuracy falls 0.93 to 0.61 swapped; structured holds 0.88.
- Each fix prices intervention data or inference cost against the robustness gain.
- Report matched and swapped numbers together with the price.