Long Video and Temporal Modeling

Actions span minutes, stories span hours, and memory spans both. Long-video modeling watches past the clip: remembering what matters, streaming what cannot wait, and answering across time.

One hour of video compresses 784-fold from raw pixels to chunk memories, the budget every long-video design lives on

Why Does This Exist?

Clip models see seconds; reality runs hours. A $16$ -frame window holds half a second — enough for a punch, useless for a plot, a burglary, or a surgery. Stretching transformers naively hits the quadratic wall: an hour at $1$ fps is $3{,}600$ frames of full attention, millions of pairwise comparisons per layer. The world does not fit in context.

Long-video modeling budgets time the way compression budgets pixels. Three strategies divide the work: hierarchical memory (long-form understanding), constant-memory streaming (streaming models), and precise localization (temporal grounding) — all increasingly backed by video foundation models. This page maps the budget; each strategy gets its own.

The one idea you need first. Quadratic attention costs $O(T^2)$ comparisons for $T$ frames: double the video, quadruple the compute. Every long-video design is an answer to that curve — hierarchy (attend locally, summarize globally), recurrence (fixed state, no pairwise past), or retrieval (index everything, attend to hits only). Name which curve-tamer a paper uses and you understand its tradeoffs instantly.

Think of It Like This

A court stenographer, not a tourist's camera roll

A tourist films everything and finds nothing: hours of footage, no index, the answer buried at minute $43$ . A court stenographer writes selectively — verbatim for testimony, summary for procedure, timestamps for exhibits — producing a record searchable in seconds. Both "recorded" the trial; only one can answer questions about it.

Long-video systems stenograph. Chunk embeddings are the transcript, event summaries are the daily digest, temporal grounding is the exhibit index. Raw frames are the camera roll: complete, honest, and useless at scale. Memory design is deciding what the stenographer writes down.

How It Actually Works

Hierarchy encodes clips independently, then reasons over summaries: frames to shots to scenes to story, each level compressing $10$ – $50\times$ . Streaming keeps a fixed state updated per frame (recurrence, causal attention with KV budgets) for live feeds that never end. Retrieval indexes chunk embeddings and fetches relevant minutes at query time, spending attention only on hits. Hybrids stenograph live (streaming) and index the notes (retrieval) for later questions.

Evaluation must match the claim: benchmarks with minute-long videos cannot certify hour-long memory, whatever the brochure says.

Worked example

One hour at $1$ fps: $3{,}600$ frames. A $16$ -frame clip model needs $3{,}600/16 = 225$ chunks. Store one $768$ -dim float32 embedding per chunk: $225 \times 768 \times 4 = 691{,}200$ bytes $\approx 675$ KB. Raw pixels at $224 \times 224 \times 3$ bytes: $3{,}600 \times 150{,}528 = 541{,}900{,}800$ bytes $\approx 517$ MB.

Compression: $541{,}900{,}800 / 691{,}200 \approx 784$ . The stenographer keeps one $784$ th of the camera roll — and every design decision is what survives that cut.

Code

The point of this snippet is the budget: frames to chunks to kilobytes, and the ratio that rules design.

1frames = 3600  # one hour at 1 fps2chunks = frames // 163memory_kb = chunks * 768 * 4 / 1024  # one float32 embedding per chunk4raw_mb = frames * 224 * 224 * 3 / 1024**25print(chunks, round(memory_kb), round(raw_mb))  # -> 225 675.0 516.86print(round(raw_mb * 1024 / memory_kb))  # -> 784

Watch Out For

Memory that evicts the answer

Fixed budgets forget: FIFO buffers drop the opening scene, aggressive summarizers blur the key detail, retrievers miss the paraphrased query. The system answers fluently from what survived — confident about an incomplete record.

Probe memory directly: needle-in-haystack tests (one planted fact per hour) separate stenographers from tourists. If recall fails at the claimed horizon, the architecture is a clip model with longer brochures.

Minute benchmarks certifying hour claims

Most "long-video" leaderboards run $1$ – $10$ minute videos that fit extended contexts without any memory machinery. Topping them proves context engineering, not hour-scale understanding — then deployment at $10\times$ the length collapses.

Match evaluation to the claim: hour claims need hour videos with questions spread across the full span (not clustered in the opening). Report performance versus position, not just averages.

The Quick Version

Clip windows see seconds; hours need memory design against quadratic attention's $O(T^2)$ wall.
Hierarchy, streaming state, and retrieval are the three curve-tamers; hybrids stenograph live and index later.
The worked fixture compresses one hour $784$ -fold: $517$ MB of pixels to $675$ KB of chunk memories.
Needle-in-haystack probes test memory honestly; fluency from survivors is not recall.
Certify hour claims on hour videos with questions spread across the span.