Gen AI

Diffusion Models

A diffusion model learns to remove noise from a picture one small step at a time, so reversing that process on pure random static builds a brand-new image.

The forward process adds a little noise to an image at each step until only static remains; the trained reverse process removes it step by step, guided by a text prompt, until a coherent image emerges from that static

Why Does This Exist?

Type "a cup of coffee on a wooden table at sunrise" into a text-to-image model and seconds later you get exactly that: steam, warm light, grain in the wood. Nobody drew it, and no photograph of that specific cup exists anywhere.

Before diffusion models, the leading approach was a GAN: a generator and a discriminator in a forger-versus-detective contest until the generator fools the judge. GANs are fast, but the training is notoriously unstable, and the generator can collapse to a handful of outputs regardless of input. A VAE trains more predictably by sampling a smooth latent space, but its outputs come out blurrier, since the objective rewards hedging across plausible answers over committing to sharp detail.

Diffusion models sidestep both problems with an odd trick: instead of learning to generate an image directly, they learn to clean up a noisy one. That reframing is far easier to train, and it's what sits behind most state-of-the-art image generators today.

Think of It Like This

Restoring a photo buried under static, one pass at a time

Picture an old photograph bleached almost entirely to grey static, with just a few ghostly hints of shape left. A skilled restorer doesn't reconstruct the whole photo in one guess. They nudge the image slightly toward something more plausible, look again at the clearer result, and nudge again. Repeat hundreds of times and a coherent photograph gradually resolves out of what looked like pure noise.

A diffusion model is that restorer, except it never saw an original photo to restore — it learned the skill of "make this slightly less noisy" from millions of examples of real images being noised up, and applies that skill starting from noise that never had a real picture underneath it at all.

How It Actually Works

The forward process — destroy, not learn

Start with a real photo of the coffee cup on the wooden table at sunrise. Add a small amount of Gaussian noise to every pixel — the image looks almost the same, just slightly grainy. Add a little more to that result, and repeat a few hundred times. The image gradually dissolves: warm light, wood grain, and the cup itself disappear into what is, at the final step, indistinguishable from pure static.

This direction is pure math, not a learned skill. There's a formula for exactly how much noise to add at each step, and no neural network touches this half of the process. It needs zero training because there's nothing to learn — destroying information in a fixed, predictable way is arithmetic, not intelligence.

The reverse process — what the network actually learns

The interesting part is teaching a neural network to run this backward. Show it a noisy image plus a number saying how noisy it is, and train it to predict exactly what noise was added — equivalently, what to subtract to make the image one notch cleaner.

Training data for this is nearly free: take any real photo, pick a random noise level, add that much noise with the fixed forward formula, and you have a (noisy image, noise level, true added noise) triple with a known answer. Do this across millions of photos at every noise level and the network stops memorizing specific images — there's no reward for that — and instead becomes a general-purpose denoiser that knows what "slightly less noisy" looks like for any image, at any stage of noise.

Generation as repeated denoising from pure noise

Generation flips the whole thing around. Start from pure random noise, the shape of an image with no real photo underneath it anywhere. Feed that into the trained network for one denoising step: the result is very slightly more image-like than static, though still mostly noise. Feed that back in for another step, and repeat for the same number of steps the forward process used, each pass nudging the pixels a little further from static and closer to something coherent.

This is where the coffee cup prompt comes in. At every step, the prompt steers which direction the network nudges the noise — the same starting static, run with a different prompt, resolves into a completely different image. By the final step, random numbers have become a specific picture: steam off the mug, sunlight on the wood grain. No single step did the work; it's hundreds of small, guided corrections compounding into a photo.

Show Me the Code

The forward process only — adding Gaussian noise at increasing levels, exactly the fixed math a diffusion model needs no training for.

1import numpy as np2
3
4def add_noise(image: np.ndarray, step: int, total_steps: int) -> np.ndarray:5    """Fixed forward process: mix in more Gaussian noise as step approaches total_steps."""6    noise_level = step / total_steps           # 0.0 = clean, 1.0 = pure noise7    keep = np.sqrt(1 - noise_level)             # how much of the real image survives8    add = np.sqrt(noise_level)                  # how much fresh noise gets mixed in9    noise = np.random.default_rng(step).normal(size=image.shape)10    return keep * image + add * noise11
12
13pixels = np.full(100, 0.9)                      # stand-in for a clean image's pixels14for step in (0, 10, 25, 40, 50):15    noisy = add_noise(pixels, step, total_steps=50)16    print(f"step {step:>2}: mean={noisy.mean():.2f} std={noisy.std():.2f}")17# -> step  0: mean=0.90 std=0.0018# -> step 50: mean=-0.04 std=1.00 -- indistinguishable from pure noise

Watch Out For

Assuming the forward process is learned

It's easy to assume some network watches the image dissolve and figures out how, since everything else here involves training. It isn't. The forward process is a closed-form formula with zero trainable parameters — every training step trains only the reverse direction. If you're asking what loss function shapes the noising schedule, that's the tell you've mixed the two up.

Expecting one reverse step to produce a coherent image

Running the trained network once on pure noise and expecting a finished coffee cup is the most common misreading here. One step moves the image only slightly less noisy than it started — the network was never trained to jump straight to a clean image, only to predict one small correction at a time. Coherence is a property of the whole chain, not any single step.

The Quick Version

The forward process gradually adds Gaussian noise to a real image over many steps until it's indistinguishable from static; it's fixed math with no training involved.
The reverse process is a neural network trained to predict the noise at a given noise level, using millions of (image, noise level, added noise) examples.
Generation starts from pure random noise, no real image involved, and repeatedly applies the trained reverse step.
Each reverse step only nudges the image slightly cleaner; coherence emerges after many steps, not one.
A text prompt steers which image emerges from the same starting noise — the basis of text-to-image generation.