Streaming Video Models

Live cameras never pause for a backward pass. Streaming models watch once, remember compactly, and answer now — constant memory no matter how long the feed runs.

A 2 kilobyte state watches a 13 gigabyte day, answering live for six million times less memory

Why Does This Exist?

Offline models assume the ending exists: rewind, rewatch, attend to everything. Live feeds — surveillance, driving, video calls, surgery — never end and never repeat. Buffering hours "just in case" costs gigabytes per camera, and latency budgets measured in milliseconds forbid reprocessing. The feed moves on with or without an answer.

Streaming models watch once. A fixed-size state updates per frame (recurrence, causal attention with bounded caches), queries read the state instead of the history, and memory stays constant from minute one to day thirty. The offline sibling is long-form understanding; the budget map is long-video modeling.

The one idea you need first. Causality means frame $t$ sees only frames up to $t$ — no future peeking, ever. Enforced by causal masking (attention) or by construction (recurrence), it guarantees live-readiness: the model's answer at noon uses exactly what a deployed system would know at noon. Any future leak in training becomes a live failure in production.

Think of It Like This

A simultaneous interpreter, not a translator

A translator works from the finished book: reread chapters, cross-check terms, polish. A simultaneous interpreter hears each sentence once, keeps notes, and speaks — no rewinds, no second hearings, latency in seconds. The notes are the state: names, numbers, open clauses compressed to a pad.

Streaming models interpret. The recurrent state is the notepad; per-frame updates are the shorthand; answers are spoken from notes, never from replay. Bigger notepads remember more and slow the hand — the latency-memory trade interpreters know by feel and models know by ablation.

How It Actually Works

Per frame: encode, update state ( $s_t = f(s_{t-1}, x_t)$ with fixed-size $s$ ), optionally emit. State forms vary — LSTM-style gates, linear recurrences, RWKV-like decays, causal transformers with sliding KV caches — but all cap memory at $O(1)$ in time. Training uses truncated backprop through time or parallelized recurrences; inference is pure forward marching. Queries arrive anytime and read the current state, so question answering, alerting, and captioning share one marching pass.

Forgetting is the design variable: decay rates and write gates decide what survives the hours.

Worked example

State: $512$ floats at $4$ bytes $= 2{,}048$ bytes $\approx 2$ KB — constant, forever. Storing instead every frame at $1$ fps for $24$ h: $86{,}400$ frames $\times 150{,}528$ bytes $\approx 13.0$ GB. Ratio: $13.0 \times 10^9 / 2{,}048 \approx 6.3 \times 10^6$ .

Six million times less memory for the privilege of forgetting wisely. The $2$ KB must hold "white van entered at 9:04" and drop cloud shapes — write gates learn exactly that triage, and failures show as vanished vans, never as crashes.

Code

The point of this snippet is the constant: state size never moves while footage accumulates without bound.

1STATE_BYTES = 512 * 4  # fixed recurrent state, forever 2 KB2
3
4def stored_bytes(frames: int) -> int:5    return frames * 224 * 224 * 3  # keeping every frame6
7
8print(STATE_BYTES)  # -> 20489print(stored_bytes(86400) // 10**9)  # -> 1310print(round(stored_bytes(86400) / STATE_BYTES / 1e6, 1))  # -> 6.3

Watch Out For

State that saturates by noon

Fixed states fill: write-everything policies overwrite morning evidence with afternoon trivia, and long-horizon questions fail long before the theoretical capacity suggests. The model answers fluently from a state that evicted the answer hours ago.

Gate writes by novelty and importance (surprise-weighted updates, episodic slots for rare events), and probe with planted-fact recall across the full horizon. Saturation is silent; only recall probes reveal it.

Latency budgets that forbid the good state

Bigger states remember more but tick slower: per-frame update cost grows with state size, and $30$ fps leaves $33$ ms total. Teams that scale state for recall miss the frame deadline, dropping frames that cost more recall than the state gained.

Co-design state size with the tick budget on target hardware. Measure end-to-end (encode plus update plus query), not FLOPs — the deadline is wall-clock.

The Quick Version

Streaming models update a fixed state per frame and answer from it — $O(1)$ memory, live by construction.
The worked fixture holds $2$ KB of state against $13$ GB of stored day: $6.3$ million times leaner.
Causality is non-negotiable: training peeks become production failures.
Gate writes by importance or saturation silently evicts answers before questions arrive.
Size state against the wall-clock tick budget, not FLOPs, on deployment hardware.