FlowNet Learned Optical Flow Model

Stack two frames and let a conv net paint the motion. FlowNet proved learned flow beats hand-tuned energy on speed and large shifts.

Stacked frames pass through correlation and refinement into dense flow

Why Does This Exist?

Classical flow solves an energy per image pair, which is slow and fragile on large shifts. FlowNet exists as the first proof that a CNN can regress dense flow directly: FlowNetS stacks frames through an encoder-decoder while FlowNetC adds an explicit correlation layer that multiplies patches across frames. It unlocked real-time learned motion and the synthetic-data recipe.

Next steps are PWC-Net for pyramids and RAFT for iteration. Basics are optical flow.

Think of It Like This

Spot-the-difference with tracing overlays

Stack two spot-the-difference cartoons on a light table and slide one until shapes align. The slide per region is the flow, and a machine that has seen thousands of puzzles learns the slide at a glance.

Frames are the cartoons and correlation is the light table. The analogy stops at lighting: cartoons keep ink while real frames change exposure, which correlation must survive.

How It Actually Works

FlowNetS concatenates $I_1$ and $I_2$ into six channels and regresses flow through contracting convolutions plus expanding refinements with skip connections. FlowNetC encodes each frame separately, then correlates patches as $c(x_1, x_2) = \sum_o f_1(x_1 + o) \cdot f_2(x_2 + o)$ over a limited displacement window, letting the decoder read match scores directly. Training on FlyingChairs synthetic pairs with endpoint-error loss transfers surprisingly well to Sintel and KITTI after finetuning.

Worked numbers

A $32$ px chair shift exceeds the classical window but sits inside FlowNetC search. Sintel clean endpoint error lands near $2.7$ px for FlowNet2 stacks versus $4$ to $6$ px for tuned Horn-Schunck on large motion. Runtime drops from seconds per pair to about $0.06$ s on a GPU, a fifty-fold speedup.

Code

1def endpoint_error(u: float, v: float, gt_u: float, gt_v: float) -> float:2    return ((u - gt_u) ** 2 + (v - gt_v) ** 2) ** 0.5  # pixels3
4
5print(round(endpoint_error(5.0, 0.0, 4.0, 1.0), 2))  # -> 1.41

Watch Out For

Testing only on FlyingChairs shapes

Synthetic chairs teach large rigid shifts but not fog, blur, or thin real edges. The symptom is great synthetic scores with smeared Sintel final passes. Always finetune on Sintel and KITTI mixes before reporting or deploying.

The Quick Version

FlowNet regresses dense flow from stacked frames with a CNN.
FlowNetS stacks early; FlowNetC correlates patch features explicitly.
FlyingChairs synthetic data enables large-motion learning.
Endpoint error near $2.7$ px on Sintel clean beats classics on big shifts.
GPU runtime near $0.06$ s makes flow real-time.
Finetuning on real mixes is required for deployment.