Skip to content
AI360XpertBeta

Score-Based Generative Models

You never teach the model the odds of each image. You teach it which way is uphill toward real images at every noise level, then walk that way out of static.

A learned score arrow at each noisy point aims at the clean image, and one Langevin step follows it with fresh noise
A learned score arrow at each noisy point aims at the clean image, and one Langevin step follows it with fresh noise

Why Does This Exist?

You want a model that draws a sharp photo that never existed. The direct route is to write down the odds of every image, p(x)p(x), and train by likelihood. That route hits a wall: the odds must sum to one, so the formula carries a normalizing constant that sums over all possible images. For a small image that sum has more terms than you could ever compute.

Score modeling walks around the wall instead of through it. Take the log, then take the gradient. The constant vanishes the moment you differentiate, because the derivative of a constant is zero. What is left is the score, the direction in which the image would look a little more real. You do not need the odds. You need the arrows.

This page stays on that idea: what the score is, how denoising score matching learns it across noise levels, and how annealed Langevin sampling walks it back to an image. For the big picture of cleaning versus guessing in one shot, start with diffusion models. For the discrete step recipe with its fixed schedule, read DDPM. For doing the walk in fewer passes, read diffusion samplers.

The one idea you need first: a gradient is a list of slopes, one per input, pointing the way uphill. If ff rises steeply when you increase xx, the gradient at that spot points toward larger xx with a matching length. The full treatment with shape checks is in gradients, and the Gaussian static this method adds and removes is in normal distribution.

Think of It Like This

A marble in a dented baking tray, in the dark

Picture a wide baking tray full of dents, and a marble resting somewhere on it. You cannot see the tray. You can only slip one finger next to the marble and feel which way the metal slopes. That slope reading is the score. You nudge the marble a finger width downhill, then you give the tray a small random shake and feel again.

The mapping is tight. The tray surface stands in for log density, with deep dents where real images sit. Your finger reading stands in for the network output sθs_\theta. Each nudge is the drift half of a Langevin step, each shake is the fresh noise half that keeps the marble from lodging in a shallow dent. A tray soaked in water stands in for high noise: dents blur into one broad bowl with a gentle slope. A dry tray stands in for low noise: dents turn sharp and the slope points at fine detail.

Where it stops: you feel the true slope with your finger, while the model must learn its slopes from samples and will guess badly where it has seen nothing. And your tray has one fixed surface, while the model learns a whole family of surfaces, one per noise level, then steps down through them in order.

How It Actually Works

Write an image as a vector xx, one number per pixel. Write p(x)p(x) for the data density, the odds of each image, and log⁡p(x)\log p(x) for its log. The score is the gradient of that log with respect to the image itself:

s(x)=∇xlog⁡p(x)s(x) = \nabla_x \log p(x)

In plain words: xx is where you stand, p(x)p(x) is how much real data sits near you, log⁡p(x)\log p(x) stretches those odds so small differences stay visible, ∇x\nabla_x means differentiate with respect to the pixels, and s(x)s(x) is the resulting arrow. A network sθ(x,σ)s_\theta(x, \sigma) learns to output that arrow, where θ\theta means the network weights and σ\sigma means the current noise level. Long arrow means steep climb nearby. Short arrow means flat ground.

The score skips the impossible constant

Any density can be split into an unnormalized shape p~(x)\tilde{p}(x) you can compute and a constant ZZ you cannot, with p(x)=p~(x)/Zp(x) = \tilde{p}(x) / Z. Take logs and the division becomes a subtraction, log⁡p(x)=log⁡p~(x)−log⁡Z\log p(x) = \log \tilde{p}(x) - \log Z. Differentiate with respect to xx and the log⁡Z\log Z term drops out, because ZZ does not depend on xx. So matching the score never asks for ZZ. That is the whole trick, and it is why score matching trains calmly where likelihood training stalls.

Denoising score matching learns it at every noise level

Clean images sit on thin clusters with vast empty space between them. In empty space the true score is undefined or wild, so training only on clean data leaves the network with no answer exactly where sampling starts: pure static. The fix is to blur first. You pick noise levels σ1>σ2>⋯>σL\sigma_1 > \sigma_2 > \dots > \sigma_L, from heavy blur down to a whisper, and you train one noise conditional network across all of them.

For one clean image x0x_0, meaning the original photo, you draw a noisy version xt=x0+σzx_t = x_0 + \sigma z, where zz is fresh Gaussian noise with mean zero. The score of that blurred point given the clean one has a closed form, (x0−xt)/σ2(x_0 - x_t) / \sigma^2, the step back toward the photo divided by the variance. The loss pushes the network toward it:

L=E[∥sθ(xt,σ)−(x0−xt)/σ2∥2]L = \mathbb{E}\left[\lVert s_\theta(x_t, \sigma) - (x_0 - x_t) / \sigma^2 \rVert^2\right]

In plain words: E\mathbb{E} means average over photos, noise levels, and noise draws. ∥⋅∥2\lVert \cdot \rVert^2 means squared length of the error arrow. At large σ\sigma the target arrows are short and point at the blurry center of everything. At small σ\sigma they are long and point at crisp detail. One network learns the whole family.

Annealed Langevin dynamics walks it back to an image

Sampling runs the learned arrows in reverse order, noisiest first. At noise level σ\sigma you repeat this update, named after Langevin, several times before stepping σ\sigma down:

x←x+ε2sθ(x,σ)+ε zx \leftarrow x + \frac{\varepsilon}{2} s_\theta(x, \sigma) + \sqrt{\varepsilon}\, z

Here ε\varepsilon (epsilon) is the step size you choose, and zz is new Gaussian noise drawn at every step. The middle term is drift: half a step along the learned arrow. The last term is shake: root step fresh noise that keeps the chain varied and stops it lodging in shallow dips. Turn ε\varepsilon down as σ\sigma falls, so early steps roam and late steps settle. Run enough steps and static sharpens into a sample. Diffusion samplers keep this walk but take longer smarter strides, and flow matching learns a straight line velocity instead of a gradient field.

The trade is plain. You buy stable supervised training and sharp varied samples, because every target is known and the injected noise keeps variety alive. You pay in sampling cost, because one image needs hundreds of ordered network passes, plus a schedule to tune: the σ\sigma ladder, the steps per level, and ε\varepsilon at each level. When you need one pass per image, or exact odds for a downstream decision, this is the wrong tool. Use a single pass generator for speed, or an invertible flow when you must score points exactly.

Worked example: one 1D step you can check

Shrink an image to one number so the arithmetic fits on a napkin. Clean value x0=2.0x_0 = 2.0. Noise level σ=1.0\sigma = 1.0. Noisy draw xt=0.8x_t = 0.8. True score first:

(x0−xt)/σ2=(2.0−0.8)/1.0=1.2(x_0 - x_t) / \sigma^2 = (2.0 - 0.8) / 1.0 = 1.2

Say the network predicts s=1.1s = 1.1, slightly short. Its squared error is (1.1−1.2)2=0.01(1.1 - 1.2)^2 = 0.01, the number the loss above averages.

Now take one Langevin step with ε=0.2\varepsilon = 0.2 and a fixed noise draw z=−0.2z = -0.2. Drift is (ε/2)s=0.1×1.1=0.11(\varepsilon / 2) s = 0.1 \times 1.1 = 0.11. Shake scale is 0.2≈0.447\sqrt{0.2} \approx 0.447, so shake is 0.447×(−0.2)≈−0.0890.447 \times (-0.2) \approx -0.089. Add them:

xnew=0.8+0.11−0.089=0.821x_{new} = 0.8 + 0.11 - 0.089 = 0.821

You moved from 0.80.8 to 0.8210.821, a small step toward 2.02.0. The drift aimed home. The shake pushed back a little, which is normal: one step need not march straight there, but the average drift points home while the shake keeps samples from collapsing onto one answer. The diagram above draws this exact step, plus a mirror point at 3.23.2 whose score of −1.2-1.2 aims left at the same clean value.

Code

This snippet replays the worked example so you can check the three numbers: the true score, the training error, and one Langevin step.

import math

def true_score(x_noisy: float, x_clean: float, sigma: float) -> float:    return (x_clean - x_noisy) / (sigma**2)

def langevin_step(x: float, score: float, eps: float, z: float) -> float:    drift = (eps / 2.0) * score  # learned pull toward data    shake = math.sqrt(eps) * z  # fresh noise drawn each step    return x + drift + shake

x0, xt, sigma = 2.0, 0.8, 1.0pred, eps, z = 1.1, 0.2, -0.2
print(true_score(xt, x0, sigma))  # -> 1.2print(round((pred - true_score(xt, x0, sigma)) ** 2, 4))  # -> 0.01print(round(langevin_step(xt, pred, eps, z), 3))  # -> 0.821

The prints agree with the hand arithmetic: true score 1.21.2, squared error 0.010.01, new position 0.8210.821. Change zz to another draw and you will see the point of the method: the drift stays +0.11+0.11 every time, while the landing jitters. That jitter is the variety in your samples.

Watch Out For

Clean training only, then sampling starts in static

If you train at one small σ\sigma or none at all, the network never sees empty space, so its arrows there are guesses. Sampling still starts in empty space. The symptom is easy to spot: the chain wanders, stalls in gray mush, or snaps to one dull sample while the loss on real photos looks fine.

Fix it with the ladder, not a bigger model. Train noise conditional across a geometric σ\sigma schedule from large to small, and check arrow norms on pure noise before you sample. If arrows at high σ\sigma are near zero or thrash, add noisier levels and steps there first.

Step size tuned at one level, used at all of them

A fixed ε\varepsilon that roams nicely at high noise will jump clean over fine detail at low noise. The symptom: early samples look plausible, then late steps get grainy or blow up into speckles, and shrinking the model changes nothing.

Tie ε\varepsilon to σ\sigma. Use smaller steps as σ\sigma falls, roughly in proportion to σ2\sigma^2, with a few steps per level before moving down. If late stage images fizz, halve ε\varepsilon at the lowest levels first and retest before touching anything else.

The Quick Version

  • A score is the gradient of log data density: an arrow at each image pointing toward more real looking pixels.
  • Differentiating the log odds deletes the normalizing constant, so training never computes the impossible sum.
  • Denoising score matching teaches those arrows at every noise level by aiming the network at (x0−xt)/σ2(x_0 - x_t) / \sigma^2.
  • Annealed Langevin sampling walks the arrows from static with drift plus fresh noise, stepping noise levels down in order.
  • Stable training and sharp varied samples cost hundreds of ordered network passes plus a schedule to tune.
  • In the 1D check, clean 2.02.0 and noisy 0.80.8 give true score 1.21.2, and one step with size 0.20.2 lands at 0.8210.821.