Concept-Based Explanations
Pixel heatmaps show where the model looked but not what it saw. Concept methods test human ideas like striped or metallic and report which ideas moved the decision.
Why Does This Exist?
A heatmap over a zebra says the model used the striped flank, but the operator still must guess whether the model learned stripes, savanna background, or fence bars. Pixel maps cannot name ideas.
Concept methods close that gap by testing labeled idea sets against the model's internals. They answer whether the zebra class direction aligns with a striped direction, how much a metallic concept matters for cutlery, and which ideas separate two confused classes. Use them when the audit question is what, after Vision Explainability Overview has settled where.
Think of It Like This
A wine judge naming the notes
A judge who only points at the glass tells you nothing. A good judge names blackcurrant, oak, and tannin, then scores each note's strength. Two wines from the same region can finally be told apart by their notes rather than their color.
Concept scores are those named notes for a classifier. Striped, metallic, and grassy are the blackcurrant and oak. The analogy stops at subjectivity: wine notes are debated, while concept sets here are fixed labeled image collections with measured alignment.
How It Actually Works
Collect 50 to 200 example images per concept (striped, dotted, metallic, grassy). Record the model's late-layer activations on concept sets versus random images, and fit the direction that separates them: the concept activation vector. Score a class by the directional derivative of its logit along that vector, averaged over class images. Values near 1 mean the concept pushes the class; near 0 means indifference.
Worked example
On a demo set of 120 zebra images: the striped direction scores 0.78 alignment with the zebra logit, grassy scores 0.12, and fence bars score 0.31. So stripes drive the call, fences contribute mildly, grass is noise. A second check on 120 horses gives striped 0.09, confirming the direction is class-specific rather than a generic texture bias. Report both the target and a control class, or a high score means nothing.
Watch Out For
Testing concepts the model never needed
A concept set with dramatic lighting or centered objects can score high for dataset reasons rather than model reasons. The symptom is every concept scoring above 0.6. Always include a random-image baseline and a control class, and distrust scores that never go low.
The Quick Version
- Concept methods test named ideas against model internals instead of pointing at pixels.
- The score is alignment between a concept direction and a class logit.
- On the demo set striped scores 0.78 for zebras and 0.09 for horses.
- Always report a control class and a random baseline beside every score.
- Use concepts for what questions, heatmaps for where questions.