Latent Diffusion

Why Does This Exist?

A diffusion model builds an image the slow way. It starts from pure noise and runs dozens of denoising steps, and each step is a full pass of a neural network over the thing being generated. The loop itself is covered in forward and reverse diffusion. What matters here is what every one of those passes has to chew through.

Run the loop on raw pixels and each pass pushes the whole image grid through the network: width times height times channels, fifty times over. For a 512 by 512 color image that is 786,432 values per pass, and the activations a capable network holds mid-pass can swallow most of a consumer GPU's memory. A datacenter card shrugs at that. The card in your desktop doesn't.

Latent diffusion keeps the loop and the quality but shrinks what the loop touches. Denoise a small compressed stand-in instead, then expand to full pixels exactly once. This page is about that compression trick and what it costs you. The loop lives on the diffusion pages; the model that made the trick famous lives at stable diffusion.

You need one idea going in: an autoencoder, a network that squeezes an image into a small code and rebuilds an approximation of the image from that code. If that is new, read variational autoencoders first, then come back. Nothing below makes sense without it.

Think of It Like This

How It Actually Works

Call the full image $x$ (pixels, the big grid) and its compressed code $z$ (the latent, the small grid). The encoder $E$ maps $x$ down to $z$, written $z = E(x)$, and the decoder $D$ maps back up to a reconstruction $\hat{x} = D(z)$, where the hat marks an approximation: close to $x$, never exactly $x$. The diffusion model never sees $x$. It lives entirely in $z$.

The autoencoder trains first, alone

Before any denoising exists, an autoencoder learns the small space on its own: squeeze images down, rebuild them, adjust until the rebuilds hold up. Variational autoencoders are the usual choice because their latent space is smooth, meaning nearby codes decode to nearby images, which is exactly what a denoiser needs as it walks from noise toward a clean code. Once this training finishes, the autoencoder freezes. Its weights never move again.

The diffusion model never learns to compress anything. It inherits a finished compressor and treats that compressor's output as its whole world.

Every noisy step happens in the small grid

Training goes like this: take a real image, encode it once to get $z = E(x)$, then add noise to $z$ and teach the denoiser to remove it. That is the familiar forward-and-reverse loop, except $z_t$ (the noisy latent at step $t$, with $t$ counting down from the noisiest level to the cleanest) sits wherever pixels used to sit. Sampling runs the same story backward: draw pure noise shaped like $z$, run the denoiser for the full step budget, and decode only the final clean latent, $\hat{x} = D(z_0)$, where $z_0$ is the fully denoised latent.

Pixels appear exactly twice in the whole pipeline: once going in, once coming out.

Worked example: the 48-fold saving

Take the standard setup: a $512 \times 512$ color image, which has 3 channels, compressed to a latent of $64 \times 64 \times 4$. That is an $8\times$ squeeze per side, since $512 / 64 = 8$, while the channel count grows from 3 to 4. Now count what one denoising pass processes. Pixels: $512 \times 512 \times 3 = 786{,}432$. Latent: $64 \times 64 \times 4 = 16{,}384$. Divide the two and you get $786{,}432 / 16{,}384 = 48$ exactly.

Each pass costs roughly in proportion to the values it touches, so every step is about 48 times cheaper, and that discount multiplies across all 50 steps: $786{,}432 \times 50 = 39{,}321{,}600$ pixel-values against $16{,}384 \times 50 = 819{,}200$ latent-values.

The trade is explicit: 48 times less work per step in exchange for whatever detail the frozen autoencoder dropped, which shows up as softness no denoiser can fix. So skip latent diffusion when pixel-exact detail is the product: measurement, metrology, anything where a texture shift changes the answer. The whole trick assumes the autoencoder reconstructs your images well. Feed it a domain it never saw and the failure lands silently, as blur, after you already paid for all 50 steps.

Code

The same step budget priced over both grids. The point is the ratio, and that the numbers match the hand count above.

1from dataclasses import dataclass2
3
4@dataclass5class Grid:6    height: int7    width: int8    channels: int9
10    def cells(self) -> int:11        return self.height * self.width * self.channels12
13
14def total_cost(grid: Grid, steps: int) -> int:15    """One denoising pass touches each value about once."""16    return grid.cells() * steps17
18
19pixel_grid = Grid(height=512, width=512, channels=3)20latent_grid = Grid(height=64, width=64, channels=4)21steps = 5022
23print(pixel_grid.cells())  # -> 78643224print(latent_grid.cells())  # -> 1638425print(total_cost(pixel_grid, steps))  # -> 3932160026print(total_cost(latent_grid, steps))  # -> 81920027print(pixel_grid.cells() / latent_grid.cells())  # -> 48.0

The prints agree with the hand count: 786,432 values against 16,384 per step, a ratio of 48 to 1. The step budget itself can shrink too: DDIM sampling reaches the image in far fewer passes, and it stacks with everything on this page.

Watch Out For

The Quick Version

• Latent diffusion runs the whole denoise loop on a small compressed latent, never on pixels. Pixels appear only twice: encode going in, decode coming out.
• A separate autoencoder trains first and then freezes. The diffusion model inherits its latent space and never learns compression.
• The standard 512 by 512 image becomes a 64 by 64 by 4 latent, so each of 50 steps processes 48 times fewer values.
• That per-step discount multiplies by the step count, which is what moves image generation from datacenter hardware onto one consumer GPU.
• The price is frozen-in detail loss. Anything the autoencoder discarded returns as softness that extra denoising cannot repair.

What to Read Next

• Forward and Reverse Diffusion
• Diffusion Models
• Diffusion Samplers
• ControlNet and Spatial Conditioning