Score-Based Generative Models
You never teach the model the odds of each image. You teach it which way is uphill toward real images at every noise level, then walk that way out of static.
Why Does This Exist?
You want a model that draws a sharp photo that never existed. The direct route is to write down the odds of every image, , and train by likelihood. That route hits a wall: the odds must sum to one, so the formula carries a normalizing constant that sums over all possible images. For a small image that sum has more terms than you could ever compute.
Score modeling walks around the wall instead of through it. Take the log, then take the gradient. The constant vanishes the moment you differentiate, because the derivative of a constant is zero. What is left is the score, the direction in which the image would look a little more real. You do not need the odds. You need the arrows.
This page stays on that idea: what the score is, how denoising score matching learns it across noise levels, and how annealed Langevin sampling walks it back to an image. For the big picture of cleaning versus guessing in one shot, start with diffusion models. For the discrete step recipe with its fixed schedule, read DDPM. For doing the walk in fewer passes, read diffusion samplers.
The one idea you need first: a gradient is a list of slopes, one per input, pointing the way uphill. If rises steeply when you increase , the gradient at that spot points toward larger with a matching length. The full treatment with shape checks is in gradients, and the Gaussian static this method adds and removes is in normal distribution.
Think of It Like This
A marble in a dented baking tray, in the dark
Picture a wide baking tray full of dents, and a marble resting somewhere on it. You cannot see the tray. You can only slip one finger next to the marble and feel which way the metal slopes. That slope reading is the score. You nudge the marble a finger width downhill, then you give the tray a small random shake and feel again.
The mapping is tight. The tray surface stands in for log density, with deep dents where real images sit. Your finger reading stands in for the network output . Each nudge is the drift half of a Langevin step, each shake is the fresh noise half that keeps the marble from lodging in a shallow dent. A tray soaked in water stands in for high noise: dents blur into one broad bowl with a gentle slope. A dry tray stands in for low noise: dents turn sharp and the slope points at fine detail.
Where it stops: you feel the true slope with your finger, while the model must learn its slopes from samples and will guess badly where it has seen nothing. And your tray has one fixed surface, while the model learns a whole family of surfaces, one per noise level, then steps down through them in order.
How It Actually Works
Write an image as a vector , one number per pixel. Write for the data density, the odds of each image, and for its log. The score is the gradient of that log with respect to the image itself:
In plain words: is where you stand, is how much real data sits near you, stretches those odds so small differences stay visible, means differentiate with respect to the pixels, and is the resulting arrow. A network learns to output that arrow, where means the network weights and means the current noise level. Long arrow means steep climb nearby. Short arrow means flat ground.
The score skips the impossible constant
Any density can be split into an unnormalized shape you can compute and a constant you cannot, with . Take logs and the division becomes a subtraction, . Differentiate with respect to and the term drops out, because does not depend on . So matching the score never asks for . That is the whole trick, and it is why score matching trains calmly where likelihood training stalls.
Denoising score matching learns it at every noise level
Clean images sit on thin clusters with vast empty space between them. In empty space the true score is undefined or wild, so training only on clean data leaves the network with no answer exactly where sampling starts: pure static. The fix is to blur first. You pick noise levels , from heavy blur down to a whisper, and you train one noise conditional network across all of them.
For one clean image , meaning the original photo, you draw a noisy version , where is fresh Gaussian noise with mean zero. The score of that blurred point given the clean one has a closed form, , the step back toward the photo divided by the variance. The loss pushes the network toward it:
In plain words: means average over photos, noise levels, and noise draws. means squared length of the error arrow. At large the target arrows are short and point at the blurry center of everything. At small they are long and point at crisp detail. One network learns the whole family.
Annealed Langevin dynamics walks it back to an image
Sampling runs the learned arrows in reverse order, noisiest first. At noise level you repeat this update, named after Langevin, several times before stepping down:
Here (epsilon) is the step size you choose, and is new Gaussian noise drawn at every step. The middle term is drift: half a step along the learned arrow. The last term is shake: root step fresh noise that keeps the chain varied and stops it lodging in shallow dips. Turn down as falls, so early steps roam and late steps settle. Run enough steps and static sharpens into a sample. Diffusion samplers keep this walk but take longer smarter strides, and flow matching learns a straight line velocity instead of a gradient field.
The trade is plain. You buy stable supervised training and sharp varied samples, because every target is known and the injected noise keeps variety alive. You pay in sampling cost, because one image needs hundreds of ordered network passes, plus a schedule to tune: the ladder, the steps per level, and at each level. When you need one pass per image, or exact odds for a downstream decision, this is the wrong tool. Use a single pass generator for speed, or an invertible flow when you must score points exactly.
Worked example: one 1D step you can check
Shrink an image to one number so the arithmetic fits on a napkin. Clean value . Noise level . Noisy draw . True score first:
Say the network predicts , slightly short. Its squared error is , the number the loss above averages.
Now take one Langevin step with and a fixed noise draw . Drift is . Shake scale is , so shake is . Add them:
You moved from to , a small step toward . The drift aimed home. The shake pushed back a little, which is normal: one step need not march straight there, but the average drift points home while the shake keeps samples from collapsing onto one answer. The diagram above draws this exact step, plus a mirror point at whose score of aims left at the same clean value.
Code
This snippet replays the worked example so you can check the three numbers: the true score, the training error, and one Langevin step.
import math
def true_score(x_noisy: float, x_clean: float, sigma: float) -> float: return (x_clean - x_noisy) / (sigma**2)
def langevin_step(x: float, score: float, eps: float, z: float) -> float: drift = (eps / 2.0) * score # learned pull toward data shake = math.sqrt(eps) * z # fresh noise drawn each step return x + drift + shake
x0, xt, sigma = 2.0, 0.8, 1.0pred, eps, z = 1.1, 0.2, -0.2
print(true_score(xt, x0, sigma)) # -> 1.2print(round((pred - true_score(xt, x0, sigma)) ** 2, 4)) # -> 0.01print(round(langevin_step(xt, pred, eps, z), 3)) # -> 0.821The prints agree with the hand arithmetic: true score , squared error , new position . Change to another draw and you will see the point of the method: the drift stays every time, while the landing jitters. That jitter is the variety in your samples.
Watch Out For
Clean training only, then sampling starts in static
If you train at one small or none at all, the network never sees empty space, so its arrows there are guesses. Sampling still starts in empty space. The symptom is easy to spot: the chain wanders, stalls in gray mush, or snaps to one dull sample while the loss on real photos looks fine.
Fix it with the ladder, not a bigger model. Train noise conditional across a geometric schedule from large to small, and check arrow norms on pure noise before you sample. If arrows at high are near zero or thrash, add noisier levels and steps there first.
Step size tuned at one level, used at all of them
A fixed that roams nicely at high noise will jump clean over fine detail at low noise. The symptom: early samples look plausible, then late steps get grainy or blow up into speckles, and shrinking the model changes nothing.
Tie to . Use smaller steps as falls, roughly in proportion to , with a few steps per level before moving down. If late stage images fizz, halve at the lowest levels first and retest before touching anything else.
The Quick Version
- A score is the gradient of log data density: an arrow at each image pointing toward more real looking pixels.
- Differentiating the log odds deletes the normalizing constant, so training never computes the impossible sum.
- Denoising score matching teaches those arrows at every noise level by aiming the network at .
- Annealed Langevin sampling walks the arrows from static with drift plus fresh noise, stepping noise levels down in order.
- Stable training and sharp varied samples cost hundreds of ordered network passes plus a schedule to tune.
- In the 1D check, clean and noisy give true score , and one step with size lands at .