GMFlow Transformer Flow Matching

Match every pixel to every pixel with attention, then read flow from the match. GMFlow swaps pyramids for one global transformer match.

Global attention matches all pixels before a small head reads the flow

Why Does This Exist?

Local windows in PWC-Net and iteration in RAFT still start from limited neighborhoods. GMFlow exists to match globally first: transformer self- and cross-attention enrich both frames, then a softmax over all pairs gives each pixel a distribution over matches. Large displacements become direct lookups instead of coarse-to-fine hunts.

Recurrent refinement remains RAFT; the first learned baseline is FlowNet.

Think of It Like This

Matching socks by spreading them all out

Find a sock match by spreading every sock on the bed at once instead of checking drawer by drawer. The pair pops visually because all candidates compete together.

Pixels are socks and softmax is the glance across the bed. The analogy stops at ambiguity: socks differ clearly while flat image patches match everywhere, needing refinement after the global vote.

How It Actually Works

Shared CNN features plus six transformer blocks produce descriptors $F_1$ and $F_2$ . Correlation $C = F_1 F_2^T / \sqrt{d}$ soft-maxes over frame two into match weights $M = \mathrm{softmax}(C)$ ; flow reads as expected displacement $\hat{f} = M G - G$ , where $G$ is the pixel grid. A small RAFT-style head refines the result for subpixel detail and occlusions. Bidirectional matching plus cycle consistency flags occluded pixels whose forward and backward matches disagree.

Worked numbers

A $100$ px jump is outside any $\pm 4$ window yet lands in the top softmax bin with weight $0.62$ against $0.03$ runner-up. Final Sintel clean error sits near $1.7$ px with one refinement pass versus $1.43$ px for twelve RAFT steps, trading a little accuracy for fewer iterations.

Code

1def expected_flow(weights: list[float], positions: list[float], grid: float) -> float:2    return sum(w * p for w, p in zip(weights, positions)) - grid  # softmax expectation3
4
5print(expected_flow([0.62, 0.03, 0.35], [100.0, 10.0, 95.0], 0.0))  # -> 95.55

Watch Out For

Trusting softmax peaks on blank regions

Flat patches give diffuse match distributions where the argmax is noise. The symptom is confident flow on walls and sky. Check distribution entropy and down-weight high-entropy pixels before refinement.

The Quick Version

GMFlow matches all pixel pairs with transformer attention first.
Softmax over matches turns correspondence into expected flow.
Global matching captures $100$ px shifts directly.
A light head refines subpixel detail and occlusions.
Error near $1.7$ px on Sintel clean with few iterations.
Diffuse match distributions mark unreliable regions to gate.