Video Diffusion Models

The Problem

By 2022, diffusion models had revolutionized image generation (e.g., DALL-E 2, Imagen). However, generating video was fundamentally harder. Video isn't just a stack of independent images; it requires strict temporal consistency (objects must move smoothly and physics must make sense from frame to frame). Naively training a diffusion model on 3D data (height $\times$ width $\times$ time) using full 3D convolutions is computationally intractable at high resolutions, and it fails to leverage the powerful spatial priors learned from massive image datasets.

The Idea

The authors proposed a highly effective architecture that extends a standard 2D image diffusion U-Net into a video diffusion model. Instead of replacing everything with 3D convolutions, they kept the 2D spatial layers and interleaved them with temporal attention blocks. This factorization allowed the model to process spatial details frame-by-frame, while the temporal blocks learned how to move those details through time.

How It Works

Factorized Space-Time Architecture The core backbone is a 3D U-Net, but it is built carefully to separate space and time:

1. Spatial Layers: Standard 2D convolutions and spatial attention operate on each video frame independently. If you pass a batch of frames through these layers, they don't look at each other.
2. Temporal Attention Layers: Interleaved after the spatial layers are 1D temporal attention blocks. These blocks treat each spatial pixel (e.g., the top-left pixel) as a sequence across time. They learn how that specific location changes over the course of the video.

This factorization is far more parameter-efficient than full 3D attention, and it allows the model to be jointly trained on both video and independent images (by simply masking out the temporal attention for image inputs).

Joint Image-Video Training Because the spatial layers are identical to an image model, the model can be trained on massive, high-quality image datasets to learn what things look like, while simultaneously training on video datasets to learn how things move.

Autoregressive Video Extension To generate longer videos, the model can be conditioned on previously generated frames. It generates a short chunk of video (e.g., 16 frames), takes the last few frames of that chunk as a conditioning input, and generates the next 16 frames, allowing for arbitrary-length generation.

Why It Mattered

This paper proved that the diffusion paradigm could be successfully adapted to video generation without completely reinventing the underlying architecture. By factorizing space and time, it made video generation computationally feasible and paved the way for models like Imagen Video and Sora (which later adopted similar space-time factorizations in transformer architectures).

What Came After

This architectural pattern (interleaving spatial and temporal blocks) became the standard for almost all early video generation models, including Runway Gen-1/Gen-2 and Stable Video Diffusion.