Skip to content
AI360XpertBeta

Flow-Based Models (Normalizing Flows)

A flow turns plain Gaussian noise into data through warps you can run backward. Sampling runs forward, scoring runs backward, and both are exact.

A single invertible coupling warp turns a Gaussian draw into data while keeping its exact likelihood
A single invertible coupling warp turns a Gaussian draw into data while keeping its exact likelihood

Why Does This Exist?

Every generative model has to answer two questions, and most answer only one well. Can you draw fresh samples? And can you score a new point, saying exactly how likely it is under the model? GANs draw beautifully but give you no density at all. Variational autoencoders score, but only a lower bound on the truth, which is part of why their samples look soft. Diffusion models earn their quality with dozens of steps per sample.

A normalizing flow refuses the split. One network, run forward, turns noise into data. The same network, run backward, returns the exact log likelihood of any point in a single pass. Here is the catch, and it is non-negotiable: every layer must be perfectly reversible, keep every dimension, and track how much it stretches volume.

This page covers that rule, the one layer design that makes it cheap, and a hand-computed example. If scoring itself feels fuzzy, read maximum likelihood estimation first. The volume math gets its full treatment in Jacobian and Hessian, and the method that drops the reversibility rule is flow matching.

Think of It Like This

A reversible roller with graph paper

You feed graph paper through a hand-cranked roller that stretches it unevenly. Crank forward and the neat grid warps into a wild pattern. Crank backward and the wild sheet returns to its neat grid, square for square, because you never cut, tore, or glued anything. Each distorted square traces back to exactly one plain square.

The density correction is square counting. Stretch one square to twice its area and the ink spreads half as dense. The roller keeps a logbook of every stretch factor, so each new square's ink density equals the old density adjusted by the logbook. That logbook is the Jacobian determinant: the number that converts densities between the plain sheet and the warped one.

Where it stops: real paper tears, folds out of plane, and gets trimmed to a smaller sheet. A flow permits none of that. Same size in, same size out, smooth and reversible at every crank position. The moment you compress or discard, the logbook stops balancing.

How It Actually Works

The one rule that buys exactness

Call the base draw zz (plain Gaussian noise you will sample), the data point xx, and the network ff (an invertible, differentiable map with x=f(z)x = f(z) and z=f−1(x)z = f^{-1}(x)). The change of variables formula says the data density equals the base density at the pre-image, corrected for how the reverse map stretches volume:

pX(x)=pZ(z) ∣det⁡Jf−1(x)∣,z=f−1(x)p_X(x) = p_Z(z) \, \left| \det J_{f^{-1}}(x) \right|, \quad z = f^{-1}(x)

Jf−1(x)J_{f^{-1}}(x) is the Jacobian matrix: every first-order partial derivative of the reverse map at xx. det⁡\det is its determinant, a single number summarizing volume change, and ∣⋅∣|\cdot| takes the absolute value since volume is never negative. In logs, products become sums:

log⁡pX(x)=log⁡pZ(z)+log⁡∣det⁡Jf−1(x)∣\log p_X(x) = \log p_Z(z) + \log \left| \det J_{f^{-1}}(x) \right|

Stack KK warps and the corrections add: x=fK(⋯f1(z))x = f_K(\cdots f_1(z)) gives a sum of KK log determinants. Sampling draws zz and runs forward. Scoring takes xx, runs backward through f−1f^{-1}, and accumulates. Two directions, one set of weights.

The assumption is strict: ff must be differentiable, exactly invertible, and shape-preserving. Downsampling, bottlenecks, and argmax-style steps are all forbidden, because each destroys the return path.

The coupling trick that keeps it cheap

A dense Jacobian determinant costs cubic time in the dimension, which would end the story for anything bigger than a toy. The affine coupling layer (the RealNVP pattern) dodges it with a freeze-and-warp split. Cut the vector in two, x=[xa,xb]x = [x_a, x_b]. Freeze the first half and warp the second with a scale and a shift predicted from the frozen half:

ya=xa,yb=xb⊙exp⁡(s(xa))+t(xa)y_a = x_a, \quad y_b = x_b \odot \exp(s(x_a)) + t(x_a)

ss is a small scale network, tt a small shift network, ⊙\odot multiplies element by element, and exp⁡\exp keeps every scale positive so the warp never collapses or flips. Now the useful bit: ss and tt only ever run forward. To invert, you already hold ya=xay_a = x_a, so you recompute the same ss and tt and undo the warp with xb=(yb−t)⊙exp⁡(−s)x_b = (y_b - t) \odot \exp(-s). Nothing needs its own inverse.

Because yay_a copies xax_a untouched, the Jacobian is triangular, and a triangular determinant is just the product of its diagonal. In logs:

log⁡∣det⁡J∣=∑isi(xa)\log \left| \det J \right| = \sum_i s_i(x_a)

A plain sum. Alternate which half freezes each layer and every coordinate gets warped within two steps.

That is the trade-off, stated plainly. You buy exact likelihood and one-pass sampling, and you pay with full-dimensional layers plus restricted operations. When images get large or you need real compression, do not fight it: reach for latent diffusion, which compresses first and diffuses later.

Worked example: one warp, four numbers

Take a 2-D standard Gaussian base and the draw z=[0.5,−0.3]z = [0.5, -0.3]. Its log density is:

log⁡pZ(z)=−12(0.52+0.32)−log⁡(2π)=−0.17−1.838=−2.008\log p_Z(z) = -\frac{1}{2}(0.5^2 + 0.3^2) - \log(2\pi) = -0.17 - 1.838 = -2.008

Freeze the first coordinate and warp the second with fixed outputs s=0.6s = 0.6 and t=0.2t = 0.2:

y1=0.5,y2=−0.3×exp⁡(0.6)+0.2=−0.3×1.822+0.2=−0.347y_1 = 0.5, \quad y_2 = -0.3 \times \exp(0.6) + 0.2 = -0.3 \times 1.822 + 0.2 = -0.347

The forward map stretches the second axis by exp⁡(0.6)\exp(0.6), so the reverse shrinks it by exp⁡(−0.6)\exp(-0.6) and contributes −0.6-0.6 to the log density:

log⁡pY(y)=−2.008−0.6=−2.608\log p_Y(y) = -2.008 - 0.6 = -2.608

Round trip to check: (y2−t)×exp⁡(−s)=(−0.347−0.2)×0.549=−0.3(y_2 - t) \times \exp(-s) = (-0.347 - 0.2) \times 0.549 = -0.3, which recovers z2z_2. The diagram above draws these same numbers: base, warp, and scored point.

Code

The point of this snippet is the round trip: warp, score, and unwarp with the same two numbers, checking the hand computation above.

import numpy as np

def log_standard_gaussian(z: np.ndarray) -> float:    return float(-0.5 * float(z @ z) - np.log(2.0 * np.pi))

def warp(z: np.ndarray, s: float, t: float) -> tuple[np.ndarray, float]:    y = np.array([z[0], z[1] * np.exp(s) + t])    return y, s  # triangular Jacobian, so the log-det is s

def unwarp(y: np.ndarray, s: float, t: float) -> np.ndarray:    return np.array([y[0], (y[1] - t) * np.exp(-s)])

z = np.array([0.5, -0.3])y, logdet = warp(z, s=0.6, t=0.2)print(round(float(y[1]), 3))  # -> -0.347print(round(log_standard_gaussian(z), 3))  # -> -2.008print(round(log_standard_gaussian(z) - logdet, 3))  # -> -2.608print(round(float(unwarp(y, 0.6, 0.2)[1]), 3))  # -> -0.3

Those four prints match the worked example exactly: warped coordinate, base log density, scored log density, recovered coordinate.

Watch Out For

Dropping the log-det term or flipping its sign

Samples look perfect and the likelihoods are nonsense. Held-out negative log likelihood drifts upward while generations improve, or model comparison crowns a checkpoint whose densities you cannot reproduce. The cause is almost always a missing or sign-flipped volume term: scoring with the base density alone, or adding +s+s in the sampling direction instead of −s-s.

The fix is directional memory. Sampling stretches by exp⁡(s)\exp(s), so it subtracts ss; scoring shrinks by exp⁡(−s)\exp(-s), so it contributes −s-s per layer, which is the −0.6-0.6 in the example. Never report a likelihood without the round-trip check above passing first.

Squeezing the vector through a bottleneck

You add a downsampling layer or a narrow latent code and the model stops inverting: shape errors on the reverse pass, or quiet garbage densities because information was discarded. The cause is structural, not a bug in your code. Invertibility forces equal dimensions in and out of every layer, so a megapixel image means megapixel activations at every depth.

Keep dimensions constant and use squeezing plus multiscale splits to manage cost, or admit the task needs compression and switch to latent diffusion. Flows win where exact scores matter: density estimation, anomaly scoring, and scientific data. They lose where pixels pile up.

The Quick Version

  • A normalizing flow maps noise to data with an invertible network, so sampling runs forward and exact scoring runs backward.
  • The change of variables sets the density: base density at the pre-image times the reverse map's absolute Jacobian determinant.
  • An affine coupling layer freezes half the vector and scales plus shifts the rest, so its log determinant is the sum of the scale outputs.
  • Every layer keeps the same dimension, and anything that compresses or discards information breaks the inverse.
  • Flows lose to diffusion on high-resolution images because full-size layers cost too much, and they win wherever an exact likelihood matters.