Flow-Based Models (Normalizing Flows)

Why Does This Exist?

In generative AI, we generally want to do two things: generate new data that looks real, and calculate exactly how real a specific piece of data is (its likelihood or probability).

Generative Adversarial Networks (GANs) generate beautiful images but have no mathematical way to tell you the probability of an image. Variational Autoencoders (VAEs) can estimate the probability, but it's only a lower bound (an approximation), so their images tend to be blurry. Diffusion models calculate probability by simulating an SDE over hundreds of steps, which is incredibly slow.

Flow-based models (Normalizing Flows) were designed to be the mathematical holy grail: a model that can generate high-quality data and calculate the exact, perfect probability of that data in a single pass. They achieve this by strictly limiting the neural network to only use mathematical operations that can be perfectly reversed.

Think of It Like This

How It Actually Works

Flow-based models rely on a fundamental theorem from probability calculus called the Change of Variables Formula.

1. The Change of Variables Formula

If you have a simple probability distribution $p(z)$ (like noise) and you pass it through a mathematical function $f(z) = x$ to get complex data, how do you find the probability of $x$? Calculus tells us that $p(x) = p(z) \times |\det(\frac{\partial z}{\partial x})|$. The scary-looking term on the right is the determinant of the Jacobian matrix. It simply measures how much the function $f$ stretches or squishes the space. If the function stretches the space by a factor of 2, the probability density drops by half.

2. Invertible Neural Networks

To use this formula, the neural network $f$ must meet two extremely strict criteria:

1. It must be perfectly invertible: $x = f(z)$ must guarantee $z = f^{-1}(x)$. If a neural network loses information (e.g., downsampling from 64x64 to 32x32), it cannot be inverted. Flow models cannot use standard bottlenecks.
2. The Jacobian determinant must be easy to calculate: For a standard neural network with millions of parameters, calculating this determinant takes $O(N^3)$ time, which would take years to compute for a single image.

Researchers solved this by inventing specialized neural network layers (like Coupling Layers in models like RealNVP and GLOW). These layers use a clever masking trick where half the data is passed through unchanged, and the other half is modified based on the first half. This guarantees perfect invertibility and makes the Jacobian determinant mathematically trivial to compute in real-time.

3. Normalizing the Flow

You stack dozens of these reversible layers together. You start with complex data $x$, pass it backward through the inverted network $f^{-1}$, and force the output $z$ to look like a standard normal Gaussian (a bell curve). By doing this, you are "normalizing" the complex data into a simple shape. To generate a new image, you just sample random noise from the bell curve and pass it forward through $f$ to get an image.

Show Me the Code

This code demonstrates a highly simplified Affine Coupling Layer, the backbone of invertible neural networks like RealNVP. Notice how the inverse function perfectly unwinds the forward function.

1import torch2import torch.nn as nn3
4class AffineCouplingLayer(nn.Module):5    def __init__(self, channels):6        super().__init__()7        # A standard, NON-invertible neural network used to calculate scale (s) and translation (t)8        # It only operates on half the channels.9        self.net = nn.Sequential(10            nn.Linear(channels // 2, 64),11            nn.ReLU(),12            nn.Linear(64, channels) # Outputs both s and t13        )14
15    def forward(self, x):16        # 1. Split the data exactly in half17        x1, x2 = x.chunk(2, dim=-1)18        19        # 2. Calculate scale and translation using only the FIRST half20        s_and_t = self.net(x1)21        s, t = s_and_t.chunk(2, dim=-1)22        23        # 3. Modify the SECOND half, leave the first half alone24        # We use torch.exp(s) to ensure the scale is strictly positive25        y1 = x126        y2 = x2 * torch.exp(s) + t27        28        # 4. Recombine29        return torch.cat([y1, y2], dim=-1)30
31    def inverse(self, y):32        # Perfect invertibility!33        y1, y2 = y.chunk(2, dim=-1)34        35        # We can calculate the exact same s and t because y1 == x136        s_and_t = self.net(y1)37        s, t = s_and_t.chunk(2, dim=-1)38        39        # Reverse the math40        x1 = y141        x2 = (y2 - t) * torch.exp(-s)42        43        return torch.cat([x1, x2], dim=-1)

Watch Out For

The Quick Version

• Normalizing Flows are generative models that can calculate the exact mathematical probability of the data they generate.
• To achieve this, the entire neural network must be built out of perfectly invertible, mathematically reversible operations.
• They rely on the Change of Variables formula to stretch and squish a simple Gaussian bell curve perfectly into a complex data distribution.
• While mathematically elegant and useful for scientific data, their inability to compress data (due to the strict invertibility requirement) makes them too computationally expensive to compete with Diffusion models for high-resolution image generation.

What to Read Next

• Read Flow Matching to see how researchers recently adapted the continuous transformation ideas from Normalizing Flows to bypass the strict invertibility requirements.
• Read Score-Based Models for the alternative stochastic math framework that led to modern diffusion.