Video Generation Architectures

Video generators choose between inflated 3D UNets, spacetime transformers, and cascades, trading memory, coherence length, and training cost.

Three video architectures trade memory, coherence, and training cost on different terms

Why Does This Exist?

Teams pick video architectures by hype and regret it in production. Inflated UNets train cheaply then cap clip length; transformers scale beautifully then devour memory; cascades render sharp frames then wobble across stages. Each failure traces to a tradeoff the chooser never weighed.

This page compares the three families on shared terms: how time enters, what caps length, and what breaks first. The short-clip mechanism is video diffusion, the long-clip exemplar is Sora-style models, and the understanding-side counterpart is video understanding.

Think of It Like This

Three bridge designs across one river

A beam bridge is cheap and proven but spans short gaps. A suspension bridge crosses wide water at high engineering cost. A pontoon bridge assembles fast from segments but sways in storms. Engineers pick by span, budget, and weather, never by fashion.

Clip length is the span, memory the budget, motion complexity the weather. Mismatched picks collapse the same way in steel and in tensors, and retrofits cost more than honest selection upfront.

How It Actually Works

Inflated 3D UNets insert temporal attention and 3D convolutions into image backbones, reusing pretrained appearance at modest extra cost; memory grows with frames times feature maps, capping clips at seconds on single cards. Spacetime transformers patchify cubes and denoise jointly, scaling cleanly to minutes with quadratic token cost. Cascaded pipelines generate small clips then super-resolve in space and time, cheap per stage but prone to cross-stage drift:

\text{UNet: } 16C + \text{time}, \qquad \text{DiT: tokens}^2, \qquad \text{cascade: stages} \times \text{small}

Autoregressive token models form a fourth lane, predicting compressed video tokens sequentially with strong coherence and slow sampling. Choice follows constraints: prototype with inflated UNets, scale length with transformers, ship resolution with cascades.

Worked example

Sixteen frames at $256$ pixels: inflated UNet about $19C$ per step with seconds-long coherence; DiT over a few thousand tokens with minute ambitions at several times the memory; cascade base at $64$ pixels plus two upsamplers, cheapest per pixel but drifting across stages. A product needing $10$ -second $720$ p clips on eight cards lands on cascades or efficient DiT, never on single-scale UNets.

Watch Out For

Benchmarking clips shorter than the product needs

Three-second metrics flatter architectures whose coherence dies at ten seconds. Teams ship on benchmark wins and hit the wall in production, where re-architecture costs months.

Evaluate at shipping length from day one. Length generalization is the metric that matters, and most papers skip it.

Ignoring the memory-length product

Attention memory grows with frames times tokens, and teams learn mid-project that target clips exceed card memory by multiples. Gradient checkpointing and windowing recover some ground at speed cost.

Budget tokens times frames before choosing. The architecture must fit the product's longest clip, not its demo.

The Quick Version

Inflated UNets reuse image backbones cheaply and cap clips at seconds.
Spacetime transformers scale to minutes with quadratic token memory.
Cascades render cheaply per stage but drift across stages.
Our $16$ -frame clip costs about $19C$ jointly versus $16C$ apart.
Evaluate at shipping length and budget memory times frames first.