CLEVR

CLEVR is a synthetic question-answering dataset built to expose whether models really reason. Every question comes from a known program, so the right answer and the right steps are both graded.

Why Does This Exist?

Real VQA datasets carry heavy answer priors. In early collections, answering "yes" to every yes-or-no question already scored surprisingly well, and question words alone predicted the answer most of the time. A model could climb the leaderboard while barely opening the image. Researchers needed a test where guessing from language alone fails by construction.

CLEVR is that test. Its scenes are rendered from a known specification and its questions are sampled from executable programs, so difficulty is controlled: attribute queries, counting, comparisons, and multi-step chains each get their own split. This page covers how the dataset is built and how to read a score on it. The general method it grades is visual reasoning; its real-photo counterpart is visual commonsense reasoning.

The one idea you need first. A functional program here is a tree of small operations: filter_color[red] keeps red objects, relate[left] keeps objects left of a reference, count returns a set's size. CLEVR samples the tree first, renders a scene that makes it answerable, then reads off the answer by executing the tree. The question text is generated from the same tree, so wording and reasoning steps always agree.

Think of It Like This

A driving exam on a closed course

A road test on public streets mixes skill with luck: traffic, weather, and the examiner's mood all leak into the score. A closed-course exam removes the luck. Every candidate faces the same cones, the same parallel-parking bay, the same emergency stop, and each maneuver is scored on its own.

CLEVR is the closed course for visual reasoning. Real photo datasets are the public streets: richer, but full of statistical luck a model can exploit. CLEVR's cones are its question families, and its per-family scores say which maneuver the model actually passed. Nobody confuses the closed course with real driving, and nobody should confuse CLEVR with real vision either.

How It Actually Works

Generation runs in four stages. First, sample a scene graph: three to ten objects, each with a shape (cube, sphere, cylinder), color, size, material, and a non-overlapping position. Second, render it with a fixed 3D engine, storing the exact scene graph alongside the pixels. Third, sample a functional program of the desired type and depth, reject it if the scene makes it ambiguous (for example a comparison with a tie), and generate the English question from templates. Fourth, execute the program on the scene graph to get the ground-truth answer.

The five question families are exist ("is there..."), count ("how many..."), compare numbers ("are there more X than Y"), compare attributes ("is the cube the same size as..."), and query attributes ("what color is..."). Strong scores require all five: models that only memorize attribute correlations pass query questions and collapse on comparisons.

Worked example

Scene holds a red cube at $(1, 1)$ , a blue cube at $(4, 1)$ , and a red sphere at $(2, 3)$ . Program for "how many red things are left of the blue cube":

filter_color[red] over $\{A, B, C\}$ keeps $\{A, C\}$ : $3$ objects in, $2$ out.
relate[left of B] with $B$ at $x = 4$ keeps objects with $x < 4$ : $A$ at $x = 1$ and $C$ at $x = 2$ both qualify, so $\{A, C\}$ survives unchanged.
count returns $2$ .

The stored answer is $2$ , and a model that outputs $2$ via a different route (say, always answering $2$ to counting questions) is caught by sibling questions on the same scene whose answers differ.

Code

The point of this snippet is the program executor: three operations over a scene list reproduce the worked example, showing why the answer is $2$ and not a guess.

1from typing import NamedTuple2
3
4class Obj(NamedTuple):5    name: str6    color: str7    x: int8
9
10scene = [Obj("A", "red", 1), Obj("B", "blue", 4), Obj("C", "red", 2)]11
12
13def filter_color(objs: list, color: str) -> list:14    return [o for o in objs if o.color == color]15
16
17def relate_left(objs: list, ref_x: int) -> list:18    return [o for o in objs if o.x < ref_x]19
20
21red = filter_color(scene, "red")22print([o.name for o in red])  # -> ['A', 'C']23print(len(relate_left(red, 4)))  # -> 2

Watch Out For

Reading a CLEVR score as general reasoning ability

CLEVR scenes contain three shapes in eight colors on a gray background. A model can master every question family while understanding nothing about texture, occlusion, lighting, or real object categories. High CLEVR accuracy with poor real-photo performance is the expected outcome, not a paradox.

Treat CLEVR as a unit test for compositional machinery: it tells you whether the reasoning steps work, given perfect perception. Pair it with a real-image benchmark before claiming anything about vision in the wild.

Training on the templates instead of the task

CLEVR questions come from templates, so their phrasing is far more regular than human questions. Models that memorize template patterns score well while failing rephrased versions of the same programs.

Check robustness by paraphrasing: if accuracy drops when "how many" becomes "give the number of" for an identical program, the model learned the template distribution, not the reasoning operation.

The Quick Version

CLEVR pairs rendered 3D scenes with questions sampled from executable programs, so answers and reasoning steps are both known.
It exists because real VQA datasets let models score well by exploiting language priors instead of looking.
Five families grade different skills: exist, count, compare numbers, compare attributes, query attributes.
The worked fixture executes filter, relate, and count over three objects to get $2$ .
A high score proves the compositional machinery works given clean perception, not that the model handles real photos.