Forward and Reverse Diffusion

Why Does This Exist?

Diffusion models pull off a strange trick. They start from pure static and land on a sharp photo that never existed. Training stays calm because it is plain regression, yet sampling takes hundreds of sequential network calls. That trade is the whole deal: stable training bought with slow generation.

Saying "add noise, then remove it" hides the two questions you need answered before the code makes sense. How much noise goes in at each step? And what does the network actually output at one reverse step, the clean image or something smaller? Get those wrong and the training loss looks like a bug and the sampler looks like magic.

This page owns those two answers: the schedule that sets the per-step noise, and the noise target the network predicts. The full DDPM recipe lives in DDPM, the faster walking plans live in diffusion samplers, and the compressed-space version lives in latent diffusion.

The one idea you need first: Gaussian noise, the static this method adds and removes. Each draw is independent randomness with mean zero, added per pixel. The full picture is in normal distribution.

Think of It Like This

How It Actually Works

The method has two directions with opposite jobs. The forward process destroys structure on a fixed plan. The reverse process learns to undo one step of that plan. Nothing learns in the forward direction at all.

The forward step adds a known amount of noise

Write $x_0$ for the clean image and $x_t$ for its noisy version at step $t$. At each step you mix in fresh Gaussian noise $\epsilon$, which means independent randomness with mean zero and variance one, drawn new every time. How much goes in is set by $\beta_t$, the noise amount at step $t$, a small positive number. Define $\alpha_t = 1 - \beta_t$, the signal kept at that step. The one-step rule is:

In plain words: shrink yesterday's image a little, then add a little static. Because every step is Gaussian, you can also jump straight from the clean image to any step. Define $\bar{\alpha}_t$, pronounced "alpha bar", as the product of everything kept so far: $\bar{\alpha}_t = \alpha_1 \alpha_2 \cdots \alpha_t$, the cumulative surviving signal. Then:

The list $\beta_1, \ldots, \beta_T$ is the noise schedule, fixed before training starts. Its shape decides where the hard work lands. A schedule that adds noise fast early destroys structure quickly, so the first reverse steps face the coarsest question (is this even a face?) while late steps polish texture. Nobody learns the schedule here. You pick it, and picking badly shows up as blurry shapes or gritty detail.

The reverse step predicts the noise, then subtracts it

The network never guesses the clean image directly. It takes the noisy image $x_t$ plus the step number $t$ and outputs $\hat{\epsilon}$, pronounced "epsilon hat", its prediction of the noise inside. Training is convenient because you created the answer yourself: during training you sampled $\epsilon$ to build $x_t$, so the loss is plain regression between the true noise and the guess:

That is the mean squared error over the noise vectors. One step of denoising then turns the prediction into $\mu$, the mean of the previous, slightly cleaner image:

You scale the current image up slightly and subtract a scaled copy of the predicted noise. Sampling sets $x_{t-1} = \mu + \sigma_t z$, where $\sigma_t$ is the reverse step's standard deviation from the schedule and $z$ is a fresh Gaussian draw, fresh randomness like $\epsilon$. The last step skips that fresh noise and keeps $\mu$ as the finished image.

Why predict noise instead of pixels? Early in the chain almost no signal remains, so guessing the whole photo is hopeless while guessing the static is still well defined. The target is known exactly at training time, which keeps training stable.

Name the trade straight. You get stable supervised training and varied samples, and you pay with sequential sampling: each image needs one network call per visited step, each waiting on the last. If you need one forward pass per image, this chain is the wrong tool. Use a single-pass generator or a distilled few-step model such as consistency models. The standing assumption is Gaussian noise on a fixed Markov chain, meaning each step depends only on the last. Break it with a schedule tuned for tiny thumbnails reused on large photos, and structure dies too fast to rebuild.

Worked example: one step out and one step back

Take one pixel value $x_0 = 1.0$ and a single step with $\beta_1 = 0.36$. Then $\alpha_1 = 1 - 0.36 = 0.64$, and since this is the first step $\bar{\alpha}_1 = 0.64$ too. The scalings are $\sqrt{0.64} = 0.8$ for the signal and $\sqrt{0.36} = 0.6$ for the noise. Draw $\epsilon = 0.5$:

Clean $1.0$ becomes noisy $1.1$. That is the diagram's middle box.

Now suppose the trained network predicts $\hat{\epsilon} = 0.45$, close to the true $0.5$. The loss is $(0.5 - 0.45)^2 = 0.0025$. The reverse mean reuses the same schedule numbers. First the correction scale: $\beta_1 / \sqrt{1 - \bar{\alpha}_1} = 0.36 / 0.6 = 0.6$. Then:

So one reverse step moves $1.1$ back to about $1.04$, nearly home to $1.0$. The error left over ($1.0375$ versus $1.0$) comes straight from the prediction gap ($0.45$ versus $0.5$). Better noise guesses mean cleaner steps, which is why the loss above is the entire training signal.

Code

The snippet below replays that exact step. It is the mechanism, not the formula retyped: corrupt once, score the prediction, walk one step back.

1import math2
3
4def forward_step(x0: float, eps: float, beta: float) -> float:5    alpha: float = 1.0 - beta6    return math.sqrt(alpha) * x0 + math.sqrt(beta) * eps7
8
9def reverse_mean(x_t: float, eps_hat: float, beta: float, alpha_bar: float) -> float:10    alpha: float = 1.0 - beta11    correction: float = beta / math.sqrt(1.0 - alpha_bar) * eps_hat12    return (x_t - correction) / math.sqrt(alpha)13
14
15x1: float = forward_step(x0=1.0, eps=0.5, beta=0.36)16print(round(x1, 4))  # -> 1.117
18mu: float = reverse_mean(x_t=x1, eps_hat=0.45, beta=0.36, alpha_bar=0.64)19print(round(mu, 4))  # -> 1.037520
21loss: float = (0.5 - 0.45) ** 222print(round(loss, 4))  # -> 0.0025

The prints match the hand computation: noisy $1.1$, one step back to $1.0375$, loss $0.0025$. Change eps_hat to $0.5$ and mu lands exactly on $1.0$, which shows the loss is the only thing steering sample quality.

Watch Out For

The Quick Version

• Forward diffusion corrupts an image through fixed Gaussian steps, and the noise schedule sets how much each step adds.
• The cumulative signal $\bar{\alpha}_t$ tells you how much of the original image survives at any step.
• The network predicts the noise inside the current image, not the clean image itself.
• Each reverse step subtracts a scaled copy of that predicted noise to reach a slightly cleaner image.
• Training is regression against the very noise the forward pass added, so its label is always known.
• Sampling is slow because steps run in order, which is why fewer-step samplers exist for the same network.

What to Read Next

• Diffusion Models
• Denoising Diffusion Probabilistic Models
• Diffusion Samplers
• Latent Diffusion