Score Matching

A probability density is hard to learn, but its slope is not. Score matching learns the gradient of the log-density directly, which points uphill toward realistic images from anywhere.

Learned score arrows point uphill toward realistic images, with strength 1.5 at position 1.5 pulling back to center

Why Does This Exist?

Energy-based models define $p(x)$ up to an unknown normalizing constant, and maximum likelihood needs that constant's gradient — an integral over all images nobody can compute. Contrastive workarounds are fiddly and unstable. The field needed a way to learn the shape of the density while ignoring its height.

Score matching learns $\nabla_x \log p(x)$ — the slope, not the surface. Slopes need no normalizing constant (it differentiates away), point toward high-density regions from anywhere, and plug directly into Langevin samplers that walk uphill with noise. Diffusion models are score matching at many noise levels wearing a scheduler. The variational cousin is variational inference; the framework is probabilistic foundations.

The one idea you need first. The score of a density is the gradient of its log: $s(x) = \nabla_x \log p(x)$ . For a standard Gaussian it is simply $s(x) = -x$ : at $x = 1.5$ the score is $-1.5$ , a pull of strength $1.5$ back toward the center. Denoising score matching trains a network to predict that pull from noisy samples, using the known noise as the teacher.

Think of It Like This

The smell of bread, no map needed

Lost in an unfamiliar city, you smell fresh bread. You have no map and no address, but the smell strengthens in one direction, so you walk that way, re-sniffing at every corner until you reach the bakery. The smell's gradient guided you; its absolute strength never mattered.

The score is that gradient. It never says how probable an image is in absolute terms (no map), only which local direction gets more realistic (stronger smell). Langevin sampling is the walk: follow the smell, add a little wandering so you visit every bakery in town rather than the nearest one forever.

The analogy stops at the nose: real score fields must be learned from data, and where no data exists the "smell" is a hallucination the sampler will still follow.

How It Actually Works

Training perturbs clean images with Gaussian noise of scale $\sigma$ and asks the network, given the noisy image, to predict the noise direction back: the target is $(x - \tilde{x}) / \sigma^2$ . Averaged over noise draws, this equals the true score of the smoothed density — no partition function anywhere. Repeating across noise levels (large $\sigma$ for global structure, small for fine detail) yields the noise-conditioned score networks inside diffusion models.

Sampling runs annealed Langevin dynamics: $x \leftarrow x + \varepsilon\, s(x) + \sqrt{2\varepsilon}\, z$ with fresh noise $z$ each step. The score climbs, the noise explores, and the schedule cools from chaos to crisp images.

Worked example

One-dimensional standard Gaussian, score $s(x) = -x$ . Start a Langevin step at $x = 1.5$ with step size $\varepsilon = 0.1$ :

Score pull: $s(1.5) = -1.5$ , drift $= 0.1 \times (-1.5) = -0.15$ .
Deterministic move: $1.5 - 0.15 = 1.35$ , closer to the center $0$ .
Noise kick: $\sqrt{0.2} \times z \approx 0.447\, z$ ; with a draw $z = 0.3$ the step lands at $1.35 + 0.134 = 1.484$ .

Repeat: every step leans centerward while the noise keeps the chain exploring instead of collapsing to the mode. That balance — pull plus jitter — is the whole sampler.

Code

The point of this snippet is the sampler in miniature: score pull plus calibrated noise reproduces the worked step.

1import math2import random3
4
5def langevin_step(x: float, eps: float, rng: random.Random) -> float:6    score = -x  # score of the standard Gaussian7    noise = math.sqrt(2.0 * eps) * rng.gauss(0.0, 1.0)8    return x + eps * score + noise  # drift plus exploration9
10
11rng = random.Random(7)  # fixed seed so the walk is checkable12print(round(langevin_step(1.5, 0.1, rng), 3))  # -> 1.484

Watch Out For

No data means invented scores

The score is learned only where noisy training samples land. Far from the data manifold — pure noise starts, adversarial corners — the network extrapolates smoothly and confidently wrong, and the sampler follows those invented arrows into garbage with high self-assurance.

Large initial noise levels exist to cover the gaps: the smoothed densities overlap and every start point has a defined pull. Skipping noise levels to "speed up" sampling strands the chain where the score is fiction.

Step size that outruns the slope

Langevin dynamics with too-large $\varepsilon$ oscillates or diverges instead of sampling: the chain jumps across the mode each step and the stationary distribution drifts from the target. Symptoms look like underfitting (blurry, washed samples) rather than the instability they are.

Tune $\varepsilon$ down until sample statistics stabilize, and keep the noise scale $\sqrt{2\varepsilon}$ matched to the drift — scaling one without the other silently changes the target distribution.

The Quick Version

Score matching learns $\nabla_x \log p(x)$ , the uphill direction, dodging the normalizing constant entirely.
Denoising score matching trains it by predicting known noise added to clean images.
The worked fixture takes a Langevin step from $1.5$ with drift $-0.15$ plus a $0.134$ noise kick.
Sampling balances score pull against noise jitter across annealed noise levels.
Scores are fiction far from training data, so keep the noise schedule covering the gaps.
Oversized Langevin steps masquerade as underfitting while sampling the wrong distribution.