Horn-Schunck Optical Flow

Horn-Schunck fills motion into every pixel, including blank walls, by demanding the flow field stay smooth and letting moving edges drag their neighbors along.

A smoothness penalty spreads motion from textured edges into blank areas until the whole frame carries one continuous flow field.

Why Does This Exist?

Lucas-Kanade tracks textured points and surrenders on blank walls. But tasks like segmentation and robot navigation need motion everywhere, including the untextured middle of a moving sheet of paper. No local window can measure motion where there is no gradient, so the answer must come from a global assumption instead.

Horn and Schunck (1981) chose smoothness: the true flow field varies slowly across the image, so each pixel's motion should resemble its neighbors'. That single prior turns an unsolvable per-pixel problem into one big solvable optimization, producing dense flow with a vector at every pixel. The price is blur across motion boundaries, which every dense method since has had to manage.

Think of It Like This

A rumor spreading through a crowd

Nobody in the middle of a packed crowd saw the stage, but everyone at the edges did, and neighbors keep comparing notes until the whole crowd agrees on what happened. The middle never observed anything directly; it inferred by staying consistent with those around it.

Horn-Schunck iterates exactly that gossip. Each pixel averages its neighbors' motion, then adjusts slightly toward its own image evidence. Blank pixels have no evidence and converge to whatever surrounds them. After enough rounds, motion has "spread" from edges across the whole frame.

How It Actually Works

1. One energy for the whole frame

Minimize data error plus smoothness penalty over all pixels:

$E = \sum \left( (I_x u + I_y v + I_t)^2 + \alpha^2 (|\nabla u|^2 + |\nabla v|^2) \right)$

$I_x, I_y, I_t$ are the image derivatives, $(u, v)$ the flow field, and $\alpha$ sets the trade: small $\alpha$ trusts image evidence (sharp but gappy), large $\alpha$ trusts smoothness (complete but blurry). There is no per-window solve; the entire field is one coupled problem.

2. Iterate local averages

The update at each pixel blends its neighbors' average $(\bar{u}, \bar{v})$ with its own constraint:

$u \leftarrow \bar{u} - \frac{I_x (I_x \bar{u} + I_y \bar{v} + I_t)}{\alpha^2 + I_x^2 + I_y^2}$

with a mirror rule for $v$ . Worked step: $I_x = 2$ , $I_y = 0$ , $I_t = -4$ , $\alpha = 1$ , neighbor average $\bar{u} = 1$ , $\bar{v} = 0$ . The constraint residual is $2 \cdot 1 + 0 - 4 = -2$ , the denominator is $1 + 4 = 5$ , so $u \leftarrow 1 - 2(-2)/5 = 1.8$ . The pixel moves from its neighbors' guess toward what its own gradient demands. Hundreds of such sweeps converge to the smooth field.

3. Dense but soft at boundaries

Because smoothness crosses object edges, a moving car smears some motion into the static road beside it. That oversmoothing is the method's signature artifact. Farneback dense flow avoids global iterations with local polynomial fits, at the cost of noisier blank regions.

Watch Out For

Alpha is the whole method

Set $\alpha$ near zero and blank regions stay chaotic; set it huge and separate motions melt into one average. There is no safe default across scenes. Sweep $\alpha$ on your own footage and watch motion boundaries specifically, because global error numbers hide boundary damage.

Occlusions violate the evidence term

Where one object covers another, pixels appear from nowhere and brightness constancy is false, yet smoothness happily invents confident flow there. Treat flow near occlusion edges as suspect, and never feed it raw into safety decisions.

The Quick Version

Horn-Schunck adds a global smoothness penalty, yielding a vector at every pixel.
$\alpha$ trades image evidence against smoothness; one worked update moves $u$ from 1.0 to 1.8.
Iteration spreads edge motion into blank regions like gossip through a crowd.
Motion boundaries smear, the signature artifact of global smoothness.
Dense and complete where Lucas-Kanade stays silent, but slower and softer at edges.