Long-Form Video Understanding

A movie is not 7,000 unrelated photos. Long-form understanding tracks characters, motives, and plot across scenes that short-clip models never connect.

A 90 minute film compresses from 1350 shots to 30 scenes to 5 acts, and questions route to the right level

Why Does This Exist?

Clip averaging destroys narrative. Mean-pooling frame features across a heist film yields "people indoors" — true, useless, and identical for a wedding in the same venue. Order, causality, and motive live between clips: the glance at minute $12$ explains the betrayal at minute $80$ , and no $16$ -frame window spans both.

Long-form understanding builds the missing spans. Hierarchical memory compresses shots into scenes into acts; character tracking carries identities across cuts; event graphs record who did what to whom. Questions route to the right level (a glance needs frames, a motive needs acts). The budget view is long-video modeling; the timestamp sibling is temporal grounding.

The one idea you need first. A story hierarchy compresses in stages: shots (seconds, camera takes), scenes (minutes, one place and time), acts (tens of minutes, narrative arcs). Each stage keeps what the next needs — shots keep appearances, scenes keep interactions, acts keep motives. Questions answer at the lowest level that suffices, so computation follows the question's scope.

Think of It Like This

Reading the novel versus shuffling its pages

Shuffle a novel's pages and every sentence survives while the story dies: characters act without motive, clues follow solutions, endings precede middles. A bag-of-sentences model of the book answers "what words appear" and nothing else. Reading in order builds the causal chain the questions actually ask about.

Clip-averaged video is the shuffled novel. Long-form models read in order: shots chain into scenes the way sentences chain into chapters, and the betrayal at minute $80$ stays connected to the glance at minute $12$ through the hierarchy that refused to shuffle.

How It Actually Works

Pipelines segment first (shot boundaries from color and motion discontinuities), encode shots with clip models, then aggregate: temporal transformers over shot embeddings for scenes, scene embeddings for acts, with character tracks and dialogue aligned throughout. Event graphs add structure (nodes for actors and objects, edges for interactions) for causal questions. Memory budgets cap how many shots stay live; the rest persist as summaries retrievable by timestamp or entity.

Evaluation uses narrative benchmarks (movie QA, episode recap, character-motive probes) with questions deliberately spanning scenes — single-scene questions would let clip models cheat.

Worked example

A $90$ -minute film: $90 \times 60 = 5{,}400$ seconds. Shots average $4$ s: $5{,}400 / 4 = 1{,}350$ shots. Scenes average $3$ min ( $180$ s): $5{,}400 / 180 = 30$ scenes. Acts: $5$ .

Compression per stage: $1{,}350 \to 30$ is $45\times$ ; $30 \to 5$ is $6\times$ . A motive question ("why did she leave?") searches $5$ act summaries, then descends into $1$ act's $6$ scenes, then $1$ scene's $45$ shots — about $50$ items inspected instead of $1{,}350$ . Hierarchy turns linear scans into guided descents.

Code

The point of this snippet is the descent: counts per level show why routing beats scanning.

1def story_levels(seconds: int, shot_s: int = 4, scene_s: int = 180, acts: int = 5) -> tuple:2    shots = seconds // shot_s3    scenes = seconds // scene_s4    return shots, scenes, acts5
6
7print(story_levels(5400))  # -> (1350, 30, 5)8print(1350 // 30, 30 // 5)  # -> (45, 6)

Watch Out For

Characters who change clothes and lighting

Re-identification across scenes breaks on costume changes, day-night cuts, and stunt doubles: the tracker splits one character into three or merges two into one, and every downstream motive answer inherits the confusion. Face recognition helps until profiles, masks, and crowds defeat it.

Track multimodally (face plus clothing plus voice plus narrative role) and keep identity uncertainty explicit. A motive attached to the wrong person is worse than no answer.

Scripts that substitute for watching

Many movie-QA questions answer from dialogue or plot summaries alone — models score well while watching nothing. Benchmarks that allow script-only solutions certify readers, not viewers.

Ablate the pixels: if accuracy holds with video removed, the benchmark leaks. Demand visual-necessity splits (questions provably unanswerable from text) before claiming video understanding.

The Quick Version

Long-form video tracks plot, motive, and identity across scenes clip models never connect.
The worked fixture compresses $1{,}350$ shots to $30$ scenes to $5$ acts, routing questions down the levels.
Guided descent inspects about $50$ items where scanning touches $1{,}350$ .
Character tracking must survive costume and lighting changes or motives attach wrongly.
Reject benchmarks solvable from scripts alone; demand visual-necessity splits.