Video Foundation Models

Every video task retrains eyes from scratch — wasteful when motion, objects, and time recur everywhere. Video foundation models learn once from web-scale video, then adapt to all of it.

One pretraining run plus ten cheap adaptations costs 8 times less than training ten specialists from scratch

Why Does This Exist?

Per-task video training repeats the same lessons: edges, objects, motion, tempo — relearned for every dataset at full price while labels stay scarce and short. Image backbones helped, but frozen image features miss motion entirely, and inflating them per task recovers only part of the gap. The field needed eyes that arrive educated.

Video foundation models pretrain on web-scale clips with self-supervision (masked tube prediction, video-text contrast), then adapt per task with probes, adapters, or short fine-tunes. One expensive education serves unlimited cheap specializations. The memory view is long-video modeling; the architecture is spatio-temporal transformers.

The one idea you need first. Masked tube modeling hides most spacetime patches (up to $90\%$ ) and reconstructs them: video's redundancy makes the puzzle solvable only with real motion understanding, and the high ratio keeps compute low (few visible tokens). Video-text contrastive aligns clips with narrations, buying zero-shot labels from alt-text-scale supervision.

Think of It Like This

Film school versus on-the-job training

Studios can train each crew member per production (on-the-job: slow, repeated, basics relearned on every set) or hire film-school graduates (shared education: lighting, lenses, pacing already known) and brief them per film in days. The school costs a fortune once; the briefings cost little each time.

Foundation pretraining is film school: motion, objects, and tempo learned from millions of clips. Adaptation is the per-film briefing: a probe or adapter mapping graduate knowledge to this task. Studios that retrain from scratch per production pay tuition on every shoot.

How It Actually Works

Pretraining mixes objectives: masked reconstruction for motion structure, contrastive video-text for semantics, sometimes temporal ordering and playback-speed puzzles. Tokenizers cut clips into spacetime tubes; backbones (factorized transformers) encode them; lightweight heads serve each loss. Adaptation freezes the backbone and trains probes (linear), adapters (small inserts), or full fine-tunes in ascending cost. Zero-shot classification via text prompts skips adaptation where vocabularies align.

Data curation decides quality: narrated, diverse, deduplicated clips beat raw hours — scale without curation teaches watermarks and compilations.

Worked example

Tokens per clip: $8$ frames $\times 14 \times 14$ patches $= 1{,}568$ . Mask $90\%$ : visible $= 1{,}568 \times 0.1 = 156.8 \approx 157$ tokens — the encoder prices $157$ , not $1{,}568$ .

Amortization: pretraining cost $C = 1{,}000$ GPU-hours; per-task adaptation $c = 20$ ; tasks $N = 10$ . Foundation route: $1{,}000 + 10 \times 20 = 1{,}200$ . From-scratch route: $10 \times 1{,}000 = 10{,}000$ . Ratio: $10{,}000/1{,}200 \approx 8.3$ . The school pays off at the second production and compounds after.

Code

The point of this snippet is the amortization: one tuition plus cheap briefings beats repeated schooling.

1def foundation_route(pretrain: float, adapt: float, tasks: int) -> float:2    return pretrain + tasks * adapt3
4
5def scratch_route(per_task: float, tasks: int) -> float:6    return tasks * per_task7
8
9print(foundation_route(1000.0, 20.0, 10))  # -> 1200.010print(scratch_route(1000.0, 10))  # -> 10000.011print(round(10000.0 / 1200.0, 1))  # -> 8.3

Watch Out For

Test sets hiding in the pretraining lake

Web-scale scraping ingests benchmark videos: Kinetics clips, movie trailers, viral moments all swim in the lake. Zero-shot scores then measure retrieval of seen footage, not generalization — and deduplication at billion scale is aspirational, not achieved.

Decontaminate aggressively (clip-level dedup against every evaluation set) and report overlap audits. A foundation result without contamination analysis is a closed-book exam taken open-book.

Backbones that quit past their horizon

Pretraining on $4$ -second clips bakes short attention spans: positional encodings, temporal kernels, and memory budgets all assume brevity. Adapting to hour-long or egocentric streams then fights the backbone's baked horizon instead of building on it.

Match pretraining horizons to target tasks (or pretrain multi-scale), and probe long-range performance before committing. A clip-educated backbone finetuned on movies stays clip-minded without explicit surgery.

The Quick Version

Video FMs pretrain once on web-scale clips (masked tubes, video-text contrast) and adapt cheaply per task.
The worked fixture masks to $157$ of $1{,}568$ tokens and amortizes $8.3\times$ over ten tasks.
Curation beats raw hours: narrated, diverse, deduplicated clips teach; compilations teach watermarks.
Audit pretraining overlaps with every evaluation set or zero-shot scores measure retrieval.
Clip-baked horizons resist long-form adaptation; match pretraining scale to the target.