Stable Diffusion

Stable Diffusion generates detailed images on a single GPU by denoising inside a tiny compressed latent space steered by frozen text embeddings.

Why Does This Exist?

Pixel-space diffusion produced gorgeous images on hardware nobody owned. Dozens of full-resolution network passes per image meant data-center GPUs, minute-long waits, and a research toy rather than a tool. Open, fast, cheap generation needed the cost structure broken, not the quality.

Stable Diffusion broke it with two moves: run diffusion in a frozen autoencoder's latent space, and steer with frozen CLIP text embeddings through cross-attention. The $48\times$ -per-step saving put $512$ -pixel generation on consumer cards and open weights put it everywhere. The compression trick itself is taught in latent diffusion, and the general image-generation field in image generation.

Think of It Like This

A mural painted on a postage stamp, then projected

An artist must paint a huge mural but rents a tiny studio. Solution: paint the whole composition on a postage stamp where every brushstroke is cheap, following written instructions slipped under the door, then project the finished stamp onto the wall with a perfect enlarger exactly once.

The latent grid is the stamp, the text prompt is the instruction slip, and the decoder is the projector. Quality survives because the stamp holds composition faithfully. It fails when the stamp is too small for the mural's detail, the compression limit below.

How It Actually Works

Three frozen-or-trained pieces cooperate. The VAE encoder compresses $512 \times 512 \times 3$ pixels to a $64 \times 64 \times 4$ latent once. A CLIP text encoder converts the prompt to $77$ token embeddings once. The trained UNet then denoises the latent for about $50$ steps, reading text through cross-attention at every resolution block, and the VAE decoder expands the finished latent to pixels once:

\text{cells}_{\text{pixel}} = 512^2 \times 3 = 786{,}432, \qquad \text{cells}_{\text{latent}} = 64^2 \times 4 = 16{,}384

The ratio is exactly $48$ . Every denoising step costs roughly per processed cell, so all fifty steps run at about two percent of pixel-space cost. Model lineage runs v1 at $512$ pixels, v2 at $768$ with a stronger text encoder, and SDXL at $1024$ with two text encoders and a larger UNet.

Worked example

Fifty DDIM steps on the latent grid touch $50 \times 16{,}384 = 819{,}200$ cell-visits; pixel space would touch $50 \times 786{,}432 = 39{,}321{,}600$ . Same step count, same architecture family, $48$ times less arithmetic, which is the gap between a desktop card finishing in seconds and a cluster job. Cross-attention adds the prompt cost: $77$ text tokens attending into each spatial block, small beside the denoising itself.

Code

Cell-count arithmetic that prices one generation in both spaces.

1def generation_cells(px: int = 512, latent: int = 64, steps: int = 50) -> tuple[int, int]:2    """Total cell visits for pixel-space versus latent-space generation."""3    pixel = px * px * 3 * steps4    lat = latent * latent * 4 * steps5    return pixel, lat6
7
8print(generation_cells())  # -> (39321600, 819200) a 48 fold gap

Watch Out For

Blaming the diffusion steps for VAE blur

Mushy fine detail, garbled small text, smeared fingers, often comes from the frozen autoencoder's $8\times$ compression, not from too few denoising steps. Raising steps from $50$ to $200$ cannot recover detail the latent never stored.

Diagnose by encoding and decoding a real photo with no diffusion at all. If the round trip already loses the detail, no sampler setting will bring it back.

Prompt words the text encoder never learned

The CLIP encoder freezes old vocabulary and old associations. Rare names, new slang, and precise counts fall through to nearest-known tokens, and cranking guidance only amplifies the wrong interpretation.

Test the encoder's reading by generating with the bare noun first. If the base concept is wrong, rephrase to words the encoder knows rather than raising guidance strength.

The Quick Version

Stable Diffusion denoises a $64 \times 64 \times 4$ latent, not $512 \times 512 \times 3$ pixels.
Per-step arithmetic drops $48$ -fold: $819{,}200$ versus $39{,}321{,}600$ cell-visits over $50$ steps.
Frozen CLIP embeddings steer every UNet block through cross-attention.
Lineage v1, v2, SDXL climbs $512 \to 768 \to 1024$ pixels with stronger text encoders.
VAE compression, not step count, usually causes blur; unknown prompt words need rephrasing.