Video Diffusion

Video diffusion adds time-aware layers to image denoisers so sixteen frames denoise together, keeping the same dog running instead of morphing mid-clip.

Temporal layers let sixteen frames denoise jointly, so motion stays consistent across the clip

Why Does This Exist?

Image models animate badly. Generate sixteen frames independently and the dog changes breed mid-run, backgrounds shimmer, and motion strobes. Time is not sixteen independent pictures; it is one scene evolving, and independent sampling cannot know that.

Video diffusion teaches the joint distribution. Temporal attention layers let every frame read its neighbours during denoising, so the model commits to one dog and one camera path before pixels settle. Pretrained image backbones inflate with added time layers, reusing years of image progress. The transformer-backbone version of this idea is diffusion transformers, and the product-scale story continues in video generation architectures.

Think of It Like This

Sixteen animators sharing one model sheet

Sixteen animators each draw one frame of a run cycle. Working blind, each invents their own dog and the reel flickers. Pin one model sheet to the wall, same spots, same build, same collar, and require every animator to check neighbours' frames before inking, and the run reads as one animal moving.

Temporal layers are the shared sheet plus the neighbour checks. The shoot fails when the sheet contradicts the action, a requested backflip the model sheet never learned, and frames smear through impossible poses.

How It Actually Works

Start from a pretrained image UNet or DiT and insert temporal blocks after spatial ones: attention over the time axis at each spatial position, plus optional 3D convolutions. Space and time factorise, spatial layers frozen or lightly tuned, temporal layers trained on video clips:

\text{cost} \approx F \times C_{\text{frame}} + T \times C_{\text{temporal}}, \qquad F = 16 \text{ frames}

Training mixes images and videos so appearance knowledge survives; sampling denoises all $F$ frames jointly from correlated noise so the clip starts coherent. Quality is judged by FVD, the video analogue of FID, plus flicker and morphing checks no single-frame metric sees.

Worked example

Sixteen frames at $256 \times 256$ . A frame UNet pass costs $C$ ; joint sampling costs roughly $16C$ plus temporal overhead near $3C$ , about $19C$ per step. Independent sampling also costs $16C$ but spends zero on coherence, which is why equal budgets give radically different watchability. Temporal attention at one spatial site mixes $16$ values per head, trivial beside the spatial maps, yet it carries the entire motion signal.

Code

Joint versus independent sampling cost, showing coherence is a small surcharge.

1def clip_cost(frames: int = 16, frame_cost: float = 1.0, temporal: float = 3.0) -> tuple[float, float]:2    """Independent versus joint sampling cost per denoising step."""3    independent = frames * frame_cost4    joint = frames * frame_cost + temporal5    return independent, joint6
7
8print(clip_cost())  # -> (16.0, 19.0) coherence costs 3 on top of 16

Watch Out For

Morphing motion the metric forgives

FVD averages over clips and tolerates slow identity drift that viewers instantly catch. A dog that gradually becomes another dog scores acceptably while the product is unusable.

Watch full clips at speed during evaluation, not just metric dashboards. Add identity-consistency checks across first and last frames for subject-driven use.

Camera motion the data never showed

Training clips skew static or slow-panning. Prompts demanding whip-pans, drone dives, or complex object interactions produce smeared frames because the temporal layers never learned those dynamics.

Scope prompts to the data's motion vocabulary or gather matching footage. Motion generalization lags appearance generalization by a wide margin.

The Quick Version

Video diffusion denoises all frames jointly with added temporal attention layers.
Image backbones inflate with time layers, reusing pretrained appearance knowledge.
Sixteen frames cost $16C$ independently versus about $19C$ jointly per step.
FVD judges clips, but slow morphing needs human-speed viewing to catch.
Training motion vocabulary bounds what prompts can demand.