Gen AI

Latent Diffusion

Latent diffusion runs the slow, iterative denoising process in a small compressed space instead of on raw pixels, so a single consumer GPU can handle it.

A pixel-space diffusion model runs every denoising step over the full image grid, but a latent-diffusion model compresses the image into a much smaller latent grid first, runs every step there instead, and only expands back to full pixel resolution once, at the very end

Why Does This Exist?

A diffusion model doesn't generate an image in one shot. It starts from noise and runs dozens of denoising steps, each one a full pass of a network over the entire image — the mechanism forward-and-reverse-diffusion covers in detail. What matters here is the size of what those steps actually touch.

Run that straight on pixels and the arithmetic turns brutal fast. Generate a portrait at a normal resolution on one consumer GPU, and every step has to push every pixel, in every channel, through a network big enough to know what a face looks like. That's a memory problem, not just a speed one — a single step's activations can eat most of what the card has to offer. A data-center GPU absorbs that fine. A card in someone's desktop mostly can't.

Latent diffusion exists because it found a way to keep the multi-step process and the quality, while shrinking what every step actually touches.

Think of It Like This

Vacuum-sealing a coat before you rearrange the suitcase

Packing a bulky winter coat, you could unfold and refold it inside the suitcase at every step of packing, wrestling with its full size again and again. Or you could vacuum-seal it first, squeezing it into a flat bag right at the start, and do all the rearranging on that flat bag instead. Only once you land and unzip the suitcase does the coat expand back to full size — and only once.

Latent diffusion runs the same way. The full-resolution image gets compressed once. All the step-by-step work happens on the small version. It only expands back to full size at the very end.

How It Actually Works

The autoencoder that makes the smaller space

Before any diffusion happens, a separate model has to build the smaller space to diffuse in. That model is an autoencoder: an encoder that compresses a real image into a much smaller latent representation, paired with a decoder that reconstructs a close approximation of the original from that latent. Variational autoencoders are the usual pick, since a well-behaved, continuous latent space is what the next stage needs.

This autoencoder trains first, on its own, entirely separate from the diffusion model — feed it images, have it compress and reconstruct them, adjust it until reconstructions hold up. By the time the diffusion model shows up, the autoencoder's job is already solved and frozen. Diffusion never learns compression. It just uses it.

Diffusion happens entirely in latent space

Once that autoencoder exists, the diffusion model never touches a raw pixel during its actual noising and denoising steps. A real portrait gets encoded into its latent exactly once. From there, the forward process that adds noise and the reverse process that removes it, step by step, both run entirely on that small latent — the same mechanism covered elsewhere in this band, just operating on a far smaller object.

The pixel-sized portrait doesn't reappear during those steps. Every denoising pass reads a small latent grid and writes back a slightly-less-noisy one. The image only becomes pixels again after the last reverse step finishes, when the frozen decoder runs once to expand the final latent into a full portrait.

Why the compute savings are the whole point

Every diffusion step is a full forward pass of the denoising network, and that pass costs roughly in proportion to how much it has to process. Run it over a full pixel grid and each step is expensive. Run it over a latent grid a fraction of the size, and each step gets proportionally cheaper, in both compute and memory — a saving that multiplies by however many steps the generation needs.

That's the whole reason this matters for the running example. A pixel-space model generating a portrait has to push every one of those dozens of steps through the full-resolution grid — exactly the memory budget a single consumer GPU struggles with. Move those steps onto a latent grid small enough to fit comfortably in that GPU's memory, and the generation that needed data-center hardware in pixel space finishes on a desktop card in reasonable time.

Show Me the Code

The same number of diffusion steps, priced over a full pixel grid versus a much smaller latent grid.

1from dataclasses import dataclass2
3
4@dataclass5class Grid:6    height: int7    width: int8    channels: int9
10    def cells(self) -> int:11        return self.height * self.width * self.channels12
13
14def total_cost(grid: Grid, steps: int) -> int:15    """One denoising pass costs roughly one unit of work per cell."""16    return grid.cells() * steps17
18
19pixel_grid = Grid(height=512, width=512, channels=3)20latent_grid = Grid(height=64, width=64, channels=4)21steps = 5022
23print(total_cost(pixel_grid, steps))    # -> 39_321_60024print(total_cost(latent_grid, steps))   # -> 819_20025# -> roughly 48x cheaper, for the same 50 denoising steps

Watch Out For

Assuming the autoencoder and diffusion model train together

It's easy to picture latent diffusion as one big model learning to compress and denoise at once, end to end. It isn't. The autoencoder trains first, by itself, purely to compress and reconstruct — no noise, no diffusion objective involved. Only after it's trained and frozen does the diffusion model start learning, using that fixed latent space as its entire universe. Train both jointly from scratch and the target the diffusion model denoises toward keeps shifting under it, since the autoencoder's own compression is still changing too.

Treating the compression ratio as free

Squeezing a portrait into a much smaller latent doesn't cost nothing. An autoencoder trained to compress too aggressively throws away detail — fine texture, small features, subtle shading — that the decoder can never recover, since decoding only expands what got encoded. It can't invent detail that was never captured. That loss stays invisible during the diffusion stage, since everything downstream works fine on the latent it's handed. It only shows up at the end, as blur no extra diffusion steps can fix.

The Quick Version

Diffusion models denoise over dozens of steps, and each step's cost scales with the size of the grid it processes.
A separate autoencoder trains first, and gets frozen, to compress images into a much smaller latent and decompress them back.
The diffusion process runs entirely inside that frozen latent space — pixels never appear mid-process.
The pixel-sized image exists only once at the start (encoding) and once at the end (decoding).
Smaller grid, cheaper step, times dozens of steps — the gap between finishing on a consumer GPU and needing a data-center cluster.