Video-Language QA Models

Watch frames in order and answer what happened and why. Time plus words beat single photos for actions.

Video-language models read frames in time to answer questions

Why Does This Exist?

Single photos miss causes that span seconds. Video-language nets sample $8$ to $32$ frames with temporal attention and train on QA plus captioning mixes. Video understanding and video captioning scores then reflect motion, not thumbnails.

This page covers frame-time QA only. Embodied uses are the next page.

The one idea you need first.

Frames $v_1$ through $v_T$ with $T$ near $8$ to $32$ join text $t$ through temporal plus cross attention. Loss is next-token on answers. Motion tests shuffle frames to check time use.

Think of It Like This

Replaying a spill to explain the cause

A photo shows a broken glass, but replay shows an elbow nudging it seconds before. Cause lives across frames, not in one still. Answers cite the nudge with times.

Frames are replay, attention is the eye, and times ground the tale. Where it stops: replays can rewind, while models read fixed samples once.

How It Actually Works

Sample $T$ frames, encode each with ViT $f$ , add time codes, mix with temporal transformer, then cross-attend to text. Tune on VideoInstruct plus caption mixes. Test QA accuracy plus temporal grounding IoU.

In plain words: $T$ sets motion detail and cost, time codes order frames, and QA grades reasons.

Worked example

Glass QA: $1$ frame accuracy $48$ percent, $8$ frames $67$ , $32$ frames $71$ . Shuffling $8$ frames drops to $54$ , proving $13$ points of true time use. The $19$ point single-to-video gain justifies frame cost.

Watch Out For

Thumbnail models passing as video nets

Mean pooling lets models answer from one frame while claiming motion skill. Shuffled tests show no drop. Always report shuffled-frame deltas before trusting video claims.

The Quick Version

Video-language nets read 8 to 32 frames with time codes.
QA plus captioning mixes teach reasons.
Temporal attention beats mean pooling.
Shuffled-frame tests prove motion use.
Frame count trades detail for cost.