Flow-Based Models (Normalizing Flows)
A flow turns plain Gaussian noise into data through warps you can run backward. Sampling runs forward, scoring runs backward, and both are exact.
Why Does This Exist?
Every generative model has to answer two questions, and most answer only one well. Can you draw fresh samples? And can you score a new point, saying exactly how likely it is under the model? GANs draw beautifully but give you no density at all. Variational autoencoders score, but only a lower bound on the truth, which is part of why their samples look soft. Diffusion models earn their quality with dozens of steps per sample.
A normalizing flow refuses the split. One network, run forward, turns noise into data. The same network, run backward, returns the exact log likelihood of any point in a single pass. Here is the catch, and it is non-negotiable: every layer must be perfectly reversible, keep every dimension, and track how much it stretches volume.
This page covers that rule, the one layer design that makes it cheap, and a hand-computed example. If scoring itself feels fuzzy, read maximum likelihood estimation first. The volume math gets its full treatment in Jacobian and Hessian, and the method that drops the reversibility rule is flow matching.
Think of It Like This
A reversible roller with graph paper
You feed graph paper through a hand-cranked roller that stretches it unevenly. Crank forward and the neat grid warps into a wild pattern. Crank backward and the wild sheet returns to its neat grid, square for square, because you never cut, tore, or glued anything. Each distorted square traces back to exactly one plain square.
The density correction is square counting. Stretch one square to twice its area and the ink spreads half as dense. The roller keeps a logbook of every stretch factor, so each new square's ink density equals the old density adjusted by the logbook. That logbook is the Jacobian determinant: the number that converts densities between the plain sheet and the warped one.
Where it stops: real paper tears, folds out of plane, and gets trimmed to a smaller sheet. A flow permits none of that. Same size in, same size out, smooth and reversible at every crank position. The moment you compress or discard, the logbook stops balancing.
How It Actually Works
The one rule that buys exactness
Call the base draw (plain Gaussian noise you will sample), the data point , and the network (an invertible, differentiable map with and ). The change of variables formula says the data density equals the base density at the pre-image, corrected for how the reverse map stretches volume:
is the Jacobian matrix: every first-order partial derivative of the reverse map at . is its determinant, a single number summarizing volume change, and takes the absolute value since volume is never negative. In logs, products become sums:
Stack warps and the corrections add: gives a sum of log determinants. Sampling draws and runs forward. Scoring takes , runs backward through , and accumulates. Two directions, one set of weights.
The assumption is strict: must be differentiable, exactly invertible, and shape-preserving. Downsampling, bottlenecks, and argmax-style steps are all forbidden, because each destroys the return path.
The coupling trick that keeps it cheap
A dense Jacobian determinant costs cubic time in the dimension, which would end the story for anything bigger than a toy. The affine coupling layer (the RealNVP pattern) dodges it with a freeze-and-warp split. Cut the vector in two, . Freeze the first half and warp the second with a scale and a shift predicted from the frozen half:
is a small scale network, a small shift network, multiplies element by element, and keeps every scale positive so the warp never collapses or flips. Now the useful bit: and only ever run forward. To invert, you already hold , so you recompute the same and and undo the warp with . Nothing needs its own inverse.
Because copies untouched, the Jacobian is triangular, and a triangular determinant is just the product of its diagonal. In logs:
A plain sum. Alternate which half freezes each layer and every coordinate gets warped within two steps.
That is the trade-off, stated plainly. You buy exact likelihood and one-pass sampling, and you pay with full-dimensional layers plus restricted operations. When images get large or you need real compression, do not fight it: reach for latent diffusion, which compresses first and diffuses later.
Worked example: one warp, four numbers
Take a 2-D standard Gaussian base and the draw . Its log density is:
Freeze the first coordinate and warp the second with fixed outputs and :
The forward map stretches the second axis by , so the reverse shrinks it by and contributes to the log density:
Round trip to check: , which recovers . The diagram above draws these same numbers: base, warp, and scored point.
Code
The point of this snippet is the round trip: warp, score, and unwarp with the same two numbers, checking the hand computation above.
import numpy as np
def log_standard_gaussian(z: np.ndarray) -> float: return float(-0.5 * float(z @ z) - np.log(2.0 * np.pi))
def warp(z: np.ndarray, s: float, t: float) -> tuple[np.ndarray, float]: y = np.array([z[0], z[1] * np.exp(s) + t]) return y, s # triangular Jacobian, so the log-det is s
def unwarp(y: np.ndarray, s: float, t: float) -> np.ndarray: return np.array([y[0], (y[1] - t) * np.exp(-s)])
z = np.array([0.5, -0.3])y, logdet = warp(z, s=0.6, t=0.2)print(round(float(y[1]), 3)) # -> -0.347print(round(log_standard_gaussian(z), 3)) # -> -2.008print(round(log_standard_gaussian(z) - logdet, 3)) # -> -2.608print(round(float(unwarp(y, 0.6, 0.2)[1]), 3)) # -> -0.3Those four prints match the worked example exactly: warped coordinate, base log density, scored log density, recovered coordinate.
Watch Out For
Dropping the log-det term or flipping its sign
Samples look perfect and the likelihoods are nonsense. Held-out negative log likelihood drifts upward while generations improve, or model comparison crowns a checkpoint whose densities you cannot reproduce. The cause is almost always a missing or sign-flipped volume term: scoring with the base density alone, or adding in the sampling direction instead of .
The fix is directional memory. Sampling stretches by , so it subtracts ; scoring shrinks by , so it contributes per layer, which is the in the example. Never report a likelihood without the round-trip check above passing first.
Squeezing the vector through a bottleneck
You add a downsampling layer or a narrow latent code and the model stops inverting: shape errors on the reverse pass, or quiet garbage densities because information was discarded. The cause is structural, not a bug in your code. Invertibility forces equal dimensions in and out of every layer, so a megapixel image means megapixel activations at every depth.
Keep dimensions constant and use squeezing plus multiscale splits to manage cost, or admit the task needs compression and switch to latent diffusion. Flows win where exact scores matter: density estimation, anomaly scoring, and scientific data. They lose where pixels pile up.
The Quick Version
- A normalizing flow maps noise to data with an invertible network, so sampling runs forward and exact scoring runs backward.
- The change of variables sets the density: base density at the pre-image times the reverse map's absolute Jacobian determinant.
- An affine coupling layer freezes half the vector and scales plus shifts the rest, so its log determinant is the sum of the scale outputs.
- Every layer keeps the same dimension, and anything that compresses or discards information breaks the inverse.
- Flows lose to diffusion on high-resolution images because full-size layers cost too much, and they win wherever an exact likelihood matters.