Paper breakdown

Latent Diffusion Models (LDM)

The 2021 paper that brought diffusion models to the masses by running the generative process in a compressed latent space, drastically reducing compute requirements.

Paper: High-Resolution Image Synthesis with Latent Diffusion Models

Authors: Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer · 2021

Read the paper

LDM uses a VAE to compress a high-res image into a smaller latent space, runs the heavy diffusion process there, and then decodes it back to pixel space.

The Problem

Standard diffusion models operate in "pixel space." To generate a 512x512 RGB image, the U-Net has to process a tensor of shape (3, 512, 512) at every single denoising step. This is computationally devastating. Training such models required thousands of GPUs, and generating a single image could take minutes on a consumer graphics card, making democratized access impossible.

The Idea

Researchers from LMU Munich and Runway realized that most pixels in an image contain redundant, high-frequency details. They proposed splitting the generative process into two stages. First, train an Autoencoder (VAE) to compress the image into a much smaller, dense "latent space" (e.g., from 3x512x512 pixels down to 4x64x64 latents). Second, train the heavy diffusion model entirely inside this tiny latent space. Once the latent is generated, the VAE decoder expands it back into a high-resolution pixel image.

How It Works

The Latent Diffusion architecture consists of three core components:

The Perceptual Compression (VAE): An encoder $E$ compresses the image $x$ into a latent $z = E(x)$ . The decoder $D$ reconstructs the image $\tilde{x} = D(z)$ . This is trained once and frozen.
The Latent Diffusion Model: A standard U-Net diffusion model is trained to denoise the latent variables $z$ . Because $z$ is 64x smaller than the pixel image, the U-Net runs exceptionally fast.
Cross-Attention Conditioning: To support text-to-image, the U-Net is augmented with cross-attention layers. A text encoder (like CLIP) embeds the user's prompt, and the U-Net attends to these text embeddings at every layer while denoising the latent.

Why It Mattered

This paper is the exact architecture behind Stable Diffusion. By moving the math to latent space, the authors reduced the computational cost of diffusion by an order of magnitude. It meant that a state-of-the-art text-to-image model could be trained on a reasonable academic budget and, crucially, could generate images in seconds on a consumer laptop GPU (like an RTX 3060). It single-handedly sparked the open-source generative AI revolution.

What Came After

Stability AI funded the compute to train the model on a massive scale, releasing Stable Diffusion v1.4. The architecture remained standard for years, powering SD 1.5, SDXL, and video generation models like SVD. It only recently began to be phased out in favor of Diffusion Transformers (DiT), though they still operate in latent space.