Visual Commonsense Reasoning

VCR asks models to answer hard questions about movie stills and then defend the answer. Picking the right rationale is the real test, because guessing the answer alone is easy.

VCR grades two chained choices over one still: pick the answer, then pick the rationale that justifies it

Why Does This Exist?

Detection says a man holds a gun. Captioning says a man holds a gun in a room. Neither says whether he is threatening someone, rehearsing a play, or cleaning the weapon — yet any person glancing at the still knows which one it is from posture, gaze, and the other people's reactions. That social and causal layer is invisible to recognition models.

Visual commonsense reasoning exists to grade exactly that layer. Each question demands an inference about intent, cause, or what happens next, and each correct answer must be defended by selecting the right rationale from four options. Answering alone is worth little; the two-stage score is the point. The stepwise machinery behind it is visual reasoning; the synthetic sibling benchmark is CLEVR.

The one idea you need first. Multiple choice with adversarial options means the wrong answers are written to be tempting: same length, same style, same mentioned objects as the right one. A model cannot win by matching surface statistics, because the distractors were built to share them. VCR applies this twice, once for answers and once for rationales.

Think of It Like This

A courtroom, not a quiz show

A quiz show rewards blurting the right answer fast, for any reason or none. A courtroom demands two things: the verdict, and the reasoning that gets you there. A correct verdict with incoherent reasoning loses the case, and coherent reasoning toward a guilty verdict convicts the wrong person just as surely.

VCR runs a courtroom. The Q-to-A stage is the verdict: what is happening. The QA-to-R stage is the closing argument: why that verdict follows from the visible evidence. Models that ace quiz-show benchmarks routinely lose in court, because they never had to connect their answer to anything in the frame.

How It Actually Works

Each item starts from a movie still with detected people and objects tagged inline (person 1, chair 2). Workers write a question, four answer choices, and for the correct answer, four rationale choices, with distractors refined by adversarial matching against early models. Evaluation has three numbers: Q-to-A accuracy (pick the answer), QA-to-R accuracy (given question plus correct answer, pick the rationale), and Q-to-AR, the joint score that counts an item correct only when both picks land.

Models typically encode the still's regions together with the tagged text, score each choice, and take the argmax per stage. The joint nature matters: a model at $70\%$ per stage scores about $0.7 \times 0.7 = 49\%$ joint if the stages fail independently, which is why Q-to-AR trails the parts by so much.

Worked example

Question: "Why is person 1 running?" Answer scores (logits) for the four choices:

$[1.2,\ 2.8,\ 0.5,\ 1.0]$

Exponentiate: $[e^{1.2}, e^{2.8}, e^{0.5}, e^{1.0}] \approx [3.32,\ 16.44,\ 1.65,\ 2.72]$ . Sum: $3.32 + 16.44 + 1.65 + 2.72 = 24.13$ . Probabilities: $[0.138,\ 0.681,\ 0.068,\ 0.113]$ . The model picks choice 2 with $68\%$ confidence: "person 1 is chasing the bus".

Conditioned on that answer, the rationale scores are $[0.4,\ 3.1,\ 0.9,\ 0.2]$ , giving exponentials $[1.49,\ 22.20,\ 2.46,\ 1.22]$ summing to $27.37$ and probabilities $[0.054,\ 0.811,\ 0.090,\ 0.045]$ . The model picks rationale 2 ("person 1 leans toward the departing bus while waving") at $81\%$ . Both stages land, so the item counts toward Q-to-AR.

Code

The point of this snippet is the two-stage scoring: softmax over answer logits, then softmax over rationale logits conditioned on the picked answer.

1import math2
3
4def softmax(logits: list) -> list:5    exps = [math.exp(z) for z in logits]  # unnormalized weights6    total = sum(exps)7    return [e / total for e in exps]8
9
10answer_probs = softmax([1.2, 2.8, 0.5, 1.0])11print([round(p, 3) for p in answer_probs])  # -> [0.138, 0.681, 0.068, 0.113]12rationale_probs = softmax([0.4, 3.1, 0.9, 0.2])13print([round(p, 3) for p in rationale_probs])  # -> [0.054, 0.811, 0.09, 0.045]14print(answer_probs[1] * rationale_probs[1])  # -> 0.552

The joint confidence is $0.681 \times 0.811 \approx 0.55$ : even two confident stages multiply into a modest joint score, which is why Q-to-AR is the strict number.

Watch Out For

Rationales chosen by style instead of evidence

Distractor rationales mention the same people and objects as the correct one, but early models still picked correctly by latching onto worker phrasing habits (correct rationales ran slightly longer, for instance). The score then measures authorship detection, not visual inference.

The defense is the same adversarial matching used to build the set: keep refreshing distractors against current models and re-check that accuracy falls when the image is hidden. If a blind model scores near chance, the visual evidence is doing the work.

Stills that freeze away the evidence

A single frame cannot show speed, speech, or what happened a second earlier, yet many questions hinge on exactly those. Models then guess from poses that underdetermine the answer, and the benchmark punishes them for missing information nobody could recover.

Read low QA-to-R scores on motion-heavy items as a data limit, not a model failure. Questions about intent that needs video belong on video benchmarks, where the evidence actually exists.

The Quick Version

VCR tests social and causal inference on movie stills: pick the answer, then pick the rationale that justifies it.
Three scores track the stages: Q-to-A, QA-to-R, and the joint Q-to-AR that requires both.
The worked fixture gives answer probabilities $[0.138, 0.681, 0.068, 0.113]$ and joint confidence about $0.55$ .
Refresh distractors adversarially and test blind models, or phrasing habits masquerade as reasoning.
Single stills underdetermine motion and speech, so video-dependent questions belong on video tests.