Denoising Diffusion Probabilistic Models
A DDPM turns pure static into an image by running one small denoising step hundreds of times, after learning that step from real photos buried in noise.
Why Does This Exist?
You want a model that draws a sharp photo that never existed. A GAN can do it in one pass, but training pits two networks against each other and the generator can collapse to a handful of outputs. A VAE trains calmly and keeps the full variety, yet its samples look soft because the loss rewards hedging across plausible answers.
DDPMs take a third path. You fix a forward process that buries a real photo in Gaussian noise over hundreds of steps, then you train one network to undo a single step. Training is plain regression with a known answer, so it stays stable and keeps the variety. The price is sampling. You'll run the network once per step in order, often up to 1,000 passes for one image. That trade is the whole story: stable training and varied samples in exchange for slow generation.
It helps to have the big picture first. Diffusion models show why cleaning beats guessing in one shot, and forward and reverse diffusion works through the schedule that sets how much noise each step adds. This page stays on the DDPM recipe itself: the fixed Gaussian forward chain, the noise prediction loss, and the step by step walk back. When you need fewer steps you'll leave DDPM sampling behind for diffusion samplers. When pixels get too large you'll move the same recipe into compression with latent diffusion. If you need a single forward pass, DDPM is the wrong tool.
Think of It Like This
Wiping frost off a windshield, one thin layer at a time
Picture a windshield frosting over on a cold night. Layer by layer the glass turns opaque until you cannot see the street at all. The thickening follows a fixed pattern you wrote down in advance: a little frost per minute, more later, until nothing shows through.
Now imagine someone who has practiced on thousands of frosted windshields. Hand them a frosted view plus a note saying how frosted it is, and they point to the frost that arrived in the latest layer. Scrape off just that layer and you are one step clearer. Look again, scrape again, repeat. After hundreds of scrapes the street reappears.
That practiced eye is the DDPM. The frost is Gaussian noise. The note is the step number. The skill is predicting the latest noise, not the street.
Where it stops: a real windshield has a real street behind the frost. A DDPM starts from solid frost with nothing behind it and builds a scene from statistics it learned across many photos. Real scraping is also deterministic, while DDPM sampling tosses in a little fresh randomness at each step except the last, so the same starting static can grow into different images.
How It Actually Works
A DDPM is two Markov chains running in opposite directions. The forward chain destroys with fixed math. The reverse process rebuilds with a learned network. Only the reverse has weights.
The forward chain adds known noise
Write for a clean image, meaning its pixels flattened into a vector. Write for the noisy version after steps, where runs from to and is the total step count. Write for the variance schedule, a small positive number that says how much fresh noise step mixes in. From it define , how much signal the step keeps, and , how much of the original image survives to step .
One forward step samples
In plain words: take the previous image, shrink it slightly, add Gaussian noise with variance . Here means a Gaussian distribution and means independent noise per pixel. Because Gaussians compose cleanly, you can jump straight from the clean image to any step without visiting the ones between:
or, with for true Gaussian noise drawn from ,
Nothing here learns. The schedule is fixed before training, and by step the image is for all practical purposes pure static. The schedule shape is its own tuning choice, worked through in forward and reverse diffusion, so this page takes it as given.
The network predicts the noise, not the image
Write for the network prediction, meaning the noise the network thinks is inside at step . Here means the network weights, often a U-Net that also reads a small embedding of so one network serves every noise level. Training picks a random photo, a random , and a random , builds with the shortcut above, and asks the network for . The simplified loss is
That's mean squared error between true noise and predicted noise, averaged over photos, noise draws, and steps.
Predicting noise beats predicting the clean image directly. At high noise almost no structure survives, so guessing the full photo is a wild extrapolation while naming the noise stays well scaled at every level. You could parametrize the network to output or a mix, and the math allows it, but the noise target trains more evenly and gives sharper samples in practice.
Sampling walks the chain backward
Generation starts from drawn from , pure static with no photo underneath. The learned reverse step is Gaussian with a predicted mean
and a fixed variance set from the schedule. One sampling step is
where means fresh Gaussian noise, drawn anew each step, except at where you set and take the mean. Run that from down to and static resolves into an image. Small errors don't doom you because later steps see their own input and correct course, and that's why hundreds of small corrections beat one giant guess.
This rests on an assumption. The true reversal is only close to Gaussian when each is small, so the schedule must keep steps tiny and numerous. Break that by jumping many steps at once with this same update and quality collapses. That failure is exactly what diffusion samplers fix with a different update, and it is why DDPM sampling stays slow when you keep the original rule.
Worked example with two steps
Take a one dimensional stand in for a pixel value so every number fits on one line. Let , , , and draw for the jump to .
First the survivals. . . So . Then and .
Now the noisy value:
Rounded, . That's the dot the diagram marks as noisy, with written beside it.
Suppose the network predicts . The training loss on this draw is . For the step back to , compute the mean. Here and . So
Rounded, the denoised value is . Sampling would then add on top of , except on the final step where you skip that extra noise. The code below repeats this exact arithmetic.
Code
This mirrors the worked example above, so you can check the three numbers the diagram prints: the noisy value, the loss, and the step back mean.
import math
def forward_xt(x0: float, bar_alpha: float, eps: float) -> float: return math.sqrt(bar_alpha) * x0 + math.sqrt(1.0 - bar_alpha) * eps
def reverse_mean(xt: float, alpha_t: float, beta_t: float, bar_alpha_t: float, eps_pred: float) -> float: return (xt - (beta_t / math.sqrt(1.0 - bar_alpha_t)) * eps_pred) / math.sqrt(alpha_t)
x0 = 0.8eps = 0.5bar_alpha_2 = 0.8 * 0.7 # alpha1 * alpha2xt = forward_xt(x0, bar_alpha_2, eps)print(round(xt, 2)) # -> 0.93
eps_pred = 0.45loss = (eps - eps_pred) ** 2print(round(loss, 4)) # -> 0.0025
mu = reverse_mean(xt, 0.7, 0.3, bar_alpha_2, eps_pred)print(round(mu, 2)) # -> 0.87The prints agree with the hand computation: noisy, loss, mean before the sampling noise. Change eps_pred to and the loss drops to zero with the mean shifting up, which shows how directly the loss steers each reverse step.
Watch Out For
Cutting steps without changing the sampler
You train with , then sample with steps using the same ancestral update to save time. The images come out soft and washed, with fine texture gone, and adding guidance makes it worse rather than better.
The DDPM update assumes tiny steps, so large jumps break the Gaussian reversal it relies on. Keep the DDPM update for the full chain. When you need few steps, switch the update itself to a dedicated rule from diffusion samplers instead of skipping with the old one.
Adding fresh noise on the final step
Your samples keep a faint grain even after the full chain, as if a light static was sprayed over an otherwise clean image. It's easy to read this as undertraining and run more epochs for nothing.
The cause is in the loop: you added at every including . The last step should be deterministic. Set when so the chain lands on exactly, and keep the stochastic addition only for where it belongs.
The Quick Version
- DDPMs train one network to predict the noise inside a noisy image at a known step, using plain squared error.
- The forward chain is fixed Gaussian math with no learned weights, and its shortcut jumps straight from a clean image to any step.
- Sampling starts from pure Gaussian static and applies the learned denoiser step by step, adding fresh noise except on the last step.
- Training stays stable and covers the full variety, but sampling costs one network pass per step, so few step generation needs a different sampler.
- DDPM is a poor fit when you need a single forward pass or tiny latency, and a good fit when you can pay hundreds of passes for varied and sharp samples.