Video Question Answering

Why Does This Exist?

Image QA ports to video too easily: sample a frame, answer from it, ignore time. A large share of early video benchmarks proved solvable from single frames — models scored competitively while blind to motion, order, and cause. Leaderboards certified frame classifiers as video understanders.

VideoQA exists to demand time. Its questions need multiple frames (before/after, counting across shots, motive from sequences), and strong systems retrieve relevant moments first (temporal grounding) then reason over them. The image-QA sibling is visual commonsense reasoning; the backbone is video foundation models.

The one idea you need first. Retrieve-then-read splits QA into grounding (find the seconds that matter) and reasoning (answer from those seconds). The split is inspectable: wrong answers trace to missed moments versus misread ones, and each half improves independently. End-to-end averages fuse both failure modes into one inscrutable score.

Think of It Like This

How It Actually Works

Standard systems encode the question, retrieve top moments with a grounding head, encode those densely, and fuse question-plus-moments through cross-attention into answer logits (multiple choice) or generated text (open-ended). Question types route differently: factoid questions read one moment, temporal questions order several, counting questions aggregate across many. Training balances answer loss with auxiliary grounding supervision so retrieval learns instead of freeloading on language priors.

Open-ended scoring uses language-model judges or constrained vocabularies; multiple-choice stays the reliable workhorse.

Worked example

"What did she pick up before leaving?" Choice logits: $[0.5,\ 2.1,\ 1.0,\ 0.3]$. Exponentials: $[e^{0.5}, e^{2.1}, e^{1.0}, e^{0.3}] \approx [1.649,\ 8.166,\ 2.718,\ 1.350]$. Sum: $1.649 + 8.166 + 2.718 + 1.350 = 13.883$. Probabilities: $[0.119,\ 0.588,\ 0.196,\ 0.097]$.

Choice $2$ ("keys, at $0:42$, before the door at $0:47$") wins at $0.59$ — grounded in two ordered moments, not one lucky frame. A blind model guessing from question priors would spread mass evenly and land near $0.25$; the $0.59$ peak prices the retrieved evidence.

Code

The point of this snippet is the verdict arithmetic: logits to probabilities, peak to answer.

1import math2
3
4def choose(logits: list) -> tuple:5    exps = [math.exp(z) for z in logits]6    total = sum(exps)7    probs = [e / total for e in exps]8    best = max(range(len(probs)), key=probs.__getitem__)9    return best, round(probs[best], 3)10
11
12print(choose([0.5, 2.1, 1.0, 0.3]))  # -> (1, 0.588)

Watch Out For

The Quick Version

• VideoQA answers language questions needing multi-frame evidence, not lucky frames.
• Retrieve-then-read separates moment finding from reasoning, making errors diagnosable.
• The worked fixture gives probabilities $[0.119, 0.588, 0.196, 0.097]$ with choice $2$ winning.
• Audit single-frame solvability or benchmarks reward frame guessing as understanding.
• Blind baselines expose language priors; video branches must earn their keep above them.

What to Read Next

• Temporal Grounding
• Video Foundation Models
• Visual Commonsense Reasoning