Rectified Flow
Train a flow model, watch which noise becomes which image, then train again on those matched pairs. Each round straightens the paths, so sampling needs fewer steps.
Why Does This Exist?
Flow matching trains on straight segments, yet its samples still need several solver steps. The gap is pairing. Each training batch pairs random noise with random images, so the straight segments cross. Where two segments cross, one spot carries two true velocities, and the network learns their average instead of either one. The learned field curves, a curved field punishes big strides, and you pay for step after step at sampling time.
Rectified flow closes that gap with a second round of training called reflow. You simulate the trained model to see which noise actually becomes which image, then train again on those matched pairs. Matched pairs cross far less, the field straightens, and big strides stop hurting.
This page covers that second round: the crossing problem, the reflow loop, and when one step is enough. The straight line setup itself lives on the flow matching page. If the diffusion chain behind all of this is fuzzy, start with diffusion models. The endpoint-jump shortcut is a different answer to the same cost problem, covered in consistency models, and the stepping machinery is in diffusion samplers.
Think of It Like This
Paving paths where people already walk
First snowfall. A hundred people cross a field, each taking the straight line from their gate to their building, and their footprints cross everywhere. Now you pave. Not along the straight lines people intended, but along the trails they actually walked: one clean sidewalk per trail, laid parallel so the paths never intersect. Next morning everyone walks a paved path from gate to door in one steady stride.
Noise samples are the gates, images are the buildings, the first footprints are the base model's simulated trajectories, and paving is retraining on those matched pairs. The paved stride is the one-step sample.
Where it stops: paving never moves a building. If the base model sends a gate to the wrong building, reflow paves that wrong trail beautifully. Straighten only a model whose few-step samples already arrive at the right places.
How It Actually Works
When straight lines cross
Name the pieces. is one noise sample, drawn from a simple distribution . is one data sample from the real distribution . is a time in , where means pure noise and means data. The point on the segment at time is , and its velocity is the constant difference . The network , with weights , takes a position and a time and tries to output that velocity. Training is squared error, , averaged over random pairs and times.
Now watch two pairs collide. Pair A runs from to , so . Pair B runs from to , so . At , A stands at , and B stands at . Same spot, same time, two different true arrows.
The loss at that spot is minimized by the average, . Check it: the gap to A is , squares , mean . The gap to B is , same . No single prediction can do better while both pairs share the spot, so the network files down both arrows into one stub. That stub is the blur you see in few-step samples.
Reflow, in two moves
First, simulate. Draw fresh noise, integrate the trained field from to with a careful solver, and record where each particle lands. In the toy, the averaged field carries forward one full stride: . Notice the swap. It lands on B's image, not A's. That swap is the crossing made visible: the averaged field cannot tell the two travellers apart.
Second, retrain. Treat those simulated landings as the new dataset and run the same straight line training on them. The new pairs are with velocity , and with velocity . Their midpoints are and : two different spots, no shared input, nothing to average. The network can fit both arrows exactly, and the two segments run parallel instead of crossing.
You can repeat the loop, and each round pairs points the current field already connects, so crossings shrink and the field straightens further. One honest limit: reflow only redistributes pairings. It adds no capacity, so a small network still rounds the field off, and the endpoints stay exactly as right or wrong as the base model's.
Sampling in one stride
Euler integration on a straight segment is exact. From with velocity and step size : . One network call, parked exactly on the target. That is the whole payoff.
Name the trade before you reach for it. You pay up front, simulating a dataset and training again, possibly more than once, to pay less per sample forever. Skip the second round when sampling is already cheap enough, as in a well tuned latent diffusion pipeline, or when you need exact likelihoods of your data, which belongs to flow-based models with their invertible layers.
Code
The point of this snippet is the crossing, replayed as arithmetic you can verify: two pairs share a midpoint, the learned average swaps their endpoints, and the reflowed pairs run parallel with exact one-step sampling.
import numpy as np
def velocity(x1: np.ndarray, x0: np.ndarray) -> np.ndarray: return x1 - x0
def point_at(x0: np.ndarray, x1: np.ndarray, t: float) -> np.ndarray: return (1.0 - t) * x0 + t * x1
def euler(x: np.ndarray, v: np.ndarray, h: float) -> np.ndarray: return x + h * v
a0 = np.array([0.0, 1.0])a1 = np.array([2.0, -1.0])b0 = np.array([0.0, -1.0])b1 = np.array([2.0, 1.0])vA = velocity(a1, a0) # pair A arrowvB = velocity(b1, b0) # pair B arrowmidA = point_at(a0, a1, 0.5) # A at t = 0.5midB = point_at(b0, b1, 0.5) # B at t = 0.5avg = (vA + vB) / 2.0 # what the network learns at the shared spotloss = float(np.mean((avg - vA) ** 2)) # per-pair squared error, averagedswap = euler(a0, avg, 1.0) # A under the averaged field, one full stridenew_mid = point_at(a0, swap, 0.5) # reflowed pair midpoint at t = 0.5one = euler(a0, velocity(swap, a0), 1.0) # one stride on the reflowed segmentprint(vA) # -> [ 2. -2.]print(vB) # -> [ 2. 2.]print(midA, midB) # -> [1. 0.] [1. 0.]print(avg) # -> [2. 0.]print(loss) # -> 2.0print(swap) # -> [2. 1.]print(new_mid) # -> [1. 1.]print(one) # -> [2. 1.]Both midpoints land on , which is why the network must average. The averaged stride from lands on , B's image, proving the swap. After reflow the midpoint moves to , clear of B's , and the single stride lands exactly.
Watch Out For
Reflow paves the wrong trail beautifully
Reflow keeps whatever pairing the base model produced, including its mistakes. If the base field sends some noise to the wrong image, the new dataset pairs them as ground truth and the next model learns that wrong pairing with full confidence.
The symptom is sharp, confident, wrong samples: one-step images that look clean but carry swapped attributes or mixed identities, and no amount of extra reflow rounds fixes them. Check few-step quality before you reflow, not after. If the base model's careful samples already go to the wrong places, fix the base model first. Reflow straightens paths; it does not correct destinations.
Cutting to one step while the field still curves
One exact stride needs a field that matches its segments. After a single reflow round on a hard dataset, yours likely does not, and jumping to one step brings the blur straight back.
The symptom is step-count dependent quality: smeared edges and washed out color at one step that clean up the moment you allow four or eight. Sweep the step count first and compare against a careful baseline. Prefer a midpoint or Heun step over plain Euler while the field is still curving, run another reflow round if the curve persists, and treat one step as earned, not default.
The Quick Version
- Rectified flow is flow matching trained a second time on pairs the first model simulated itself, a loop called reflow.
- Random pairing lets straight segments cross, and at each crossing the network learns the average arrow, which curves the field and forces small solver steps.
- In the running example, pairs to and to share midpoint , so the network learns with a per-pair loss of neither arrow can beat.
- Reflowed pairs and run parallel through distinct midpoints and , so one Euler stride of size lands exactly.
- You pay with an extra simulation plus retraining round to buy cheaper sampling forever, and it only pays off when sampling cost dominates.
- Never reflow a model whose careful samples already land in the wrong places, since straightening preserves destinations, mistakes included.