Visual Reasoning

Answering questions about an image takes more than naming objects. Visual reasoning chains observations into steps: find this, compare it to that, count what remains.

Why Does This Exist?

A classifier can ace visual question answering without ever looking at the picture. On early VQA datasets, models learned that questions starting with "what sport" are usually answered "tennis", and that "yes" beats "no" two to one. They were answering from language statistics, not from pixels. The scores looked great and meant little.

Visual reasoning exists to close that gap. It treats a hard question as a small program: locate the mentioned objects, apply relations and comparisons, then aggregate. Each step reads the image, so there is nowhere for a language shortcut to hide. This page covers the stepwise view and how it is executed. The datasets that grade it — CLEVR and visual commonsense reasoning — each get their own page.

The one idea you need first. An object detector outputs boxes with labels and coordinates, such as "red cube at (1, 1)". Reasoning starts where detection stops: it takes that list of grounded objects as its input and computes over it. If you have not met detectors, think of this page's input as a spreadsheet where every row is one object with its properties and position.

Think of It Like This

A detective with an evidence board

A detective does not solve a case by glancing at the crime scene once. She pins photos to a board, draws strings between the ones that connect, crosses out the ones with alibis, and counts who is left. Each move is small, checkable, and written down.

Visual reasoning works the same way. The detections are the pinned photos. Filtering by color is crossing out everyone with an alibi. The spatial relation is a string between two pins. The count at the end is only as good as the steps, and because every step is written down, you can point at exactly where a wrong answer went wrong.

The analogy stops here: the detective understands motive, and the model does not. It executes the steps; it feels nothing about them.

How It Actually Works

Most visual reasoning systems follow three stages. First, perceive: run a detector or segmenter to get objects with attributes (color, shape, size) and positions. Second, parse: turn the question into steps, either as an explicit program ("filter red", "relate left-of", "count"), as a sequence of attention glimpses (MAC networks), or as generated code executed against the scene. Third, execute: run each step against the objects and return the result.

The explicit-program form is the clearest to learn from, because every intermediate is inspectable. A question such as "how many red objects sit left of the blue cube" compiles to three operations applied in order, each consuming the previous step's object set.

Worked example

Take a tiny scene with four objects and their $(x, y)$ positions:

A is a red cube at $(1, 1)$
B: blue cube at $(4, 1)$
C is a red sphere at $(2, 3)$
D: green cylinder at $(5, 4)$

Question: "How many red objects are left of the blue cube?"

Filter red. Keep objects whose color is red: $\{A, C\}$ .
Relate left-of B. The blue cube sits at $x = 4$ , so keep objects with $x < 4$ : $A$ has $x = 1$ and $C$ has $x = 2$ , so both survive: $\{A, C\}$ .
Count. The set has $2$ members, so the answer is $2$ .

Change one fact and watch the pipeline respond: move $C$ to $x = 6$ and step 2 drops it, giving $\{A\}$ and the answer $1$ . That sensitivity to the actual layout is exactly what single-glance classifiers lack.

Code

The point of this snippet is the executor, not the parser. Given the object list, three list comprehensions reproduce the worked example, so you can test how each step changes the answer.

1from typing import NamedTuple2
3
4class Obj(NamedTuple):5    name: str6    color: str7    x: int8    y: int9
10
11scene = [12    Obj("A", "red", 1, 1),13    Obj("B", "blue", 4, 1),14    Obj("C", "red", 2, 3),15    Obj("D", "green", 5, 4),16]17
18red = [o for o in scene if o.color == "red"]  # filter step19left = [o for o in red if o.x < 4]  # relate step (left of B at x=4)20print([o.name for o in left])  # -> ['A', 'C']21print(len(left))  # -> 2

Watch Out For

A right answer from the wrong steps

Stepwise systems can still cheat. If every training question about red objects happens to have answer $2$ , the executor learns to output $2$ whenever it sees "red" and stops reading positions. The final accuracy hides this completely.

The fix is to grade the intermediates, not just the answer. Check that the filter step returns the right set and the relation step keeps the right members on held-out layouts where the correlations are broken. An answer is only evidence of reasoning when the steps that produced it are also correct.

Errors that compound down the chain

Three steps at $90\%$ accuracy each give roughly $0.9^3 \approx 73\%$ end to end. Long chains fail more often than any single step suggests, and the failure lands on the final answer with no indication of which step broke.

Keep chains short, make early steps the most reliable ones (perception before comparison), and log every intermediate set during development so the first wrong step is visible instead of the last wrong answer.

The Quick Version

Visual reasoning answers image questions with explicit steps over detected objects, not one glance at the pixels.
The standard chain is perceive (detect), parse (question into steps), execute (run steps over objects).
The worked fixture filters red objects, keeps those left of $x = 4$ , and counts $2$ .
Grade the intermediate steps on layouts that break training correlations, or shortcuts hide behind good accuracy.
Chain accuracy multiplies, so three $90\%$ steps give about $73\%$ end to end.