Farneback Optical Flow

Farneback approximates each image neighborhood as a smooth polynomial hill, then reads the motion between frames from how the hill's coefficients shifted.

Each neighborhood becomes a quadratic surface, and the displacement between the two frames' surfaces is the local motion.

Why Does This Exist?

Dense flow options in the classics were Horn-Schunck with its hundreds of global iterations and blurry boundaries, or sparse Lucas-Kanade that stays silent on most pixels. Gunnar Farneback (2003) wanted dense motion from local computation only: no global sweeps, no texture gating, a vector everywhere from neighborhood math.

The trick is polynomial expansion. If each neighborhood is a quadratic surface, then a shift of the image becomes a clean algebraic change in the coefficients, and displacement falls out of a tiny linear solve per neighborhood. OpenCV's calcOpticalFlowFarneback ships this and remains the default dense baseline people compare learned methods against.

Think of It Like This

Matching two sand dunes

Two photos show the same dune a minute apart, and the wind moved it. Instead of tracking individual grains, you fit each photo's dune with a smooth mathematical mound, then ask how far the mound's peak slid. The fit ignores grain noise; the peak shift is the motion.

Farneback fits such mounds (quadratics) to every neighborhood in both frames. Coefficient changes between the fits encode the shift directly, so motion comes from comparing six numbers, not hundreds of pixels.

How It Actually Works

1. Expand each neighborhood

Model the neighborhood around a pixel as $f(x) = x^T A x + b^T x + c$ , a quadratic with matrix $A$ , vector $b$ , and scalar $c$ fit by weighted least squares over the window. In 1D this is just fitting a parabola $ax^2 + bx + c$ to the brightness profile.

2. Shift becomes coefficient change

If frame two is frame one shifted by displacement $d$ , the quadratics relate as $A_2 = A_1$ and $b_2 = b_1 - 2 A_1 d$ . Solving gives $d$ from the measured $b$ change. Worked 1D: $A = 1$ , $b_1 = 4$ , $b_2 = 0$ , so $d = (4 - 0) / (2 \cdot 1) = 2$ pixels. The real 2D version solves a small weighted system over a larger neighborhood for stability, then optionally iterates the estimate a couple of times.

3. Pyramids carry large motion

One expansion only sees shifts within its window, so OpenCV builds a pyramid: estimate coarsely, warp, refine. The famous parameter set (pyr_scale=0.5, several levels, window ~15) is a coarse-to-fine schedule, and shrinking the window buys speed at the cost of noise.

Watch Out For

Polynomials smooth away fine motion

The quadratic fit is a low-pass filter: thin structures and small fast objects get averaged into their surroundings before motion is even estimated. If tiny objects vanish from your flow field while big ones track fine, the window is too large for them, not the scene too hard.

Parameter soup with no universal recipe

Levels, window size, iterations, and polynomial width interact, and defaults tuned for slow driving footage fail on sports video. Change one parameter at a time against visualized flow on your own clips; numeric error alone will mislead you into over smoothing.

The Quick Version

Farneback fits a quadratic to each neighborhood in both frames and reads shift from coefficient changes.
The 1D example recovers a 2-pixel shift exactly from $b_1 = 4$ , $b_2 = 0$ .
Local solves mean dense flow with no global iteration, unlike Horn-Schunck.
Pyramids extend it to large motion; window size trades noise against fine detail.
OpenCV's default dense baseline, still the first thing to try before learned flow.