Visual explainer
Optical Flow
How image motion becomes a vector field, how local gradients constrain it, and why edges, occlusion, and lighting can mislead.
1. Track Apparent Motion
Optical flow assigns image locations a displacement toward the next frame. It measures apparent motion in pixels, which can come from object movement, camera movement, or both. The same textured patch appears farther right and lower in the second frame.
2. Measure The Displacement
Subtract corresponding coordinates: dx = 112 − 100 = 12, dy = 84 − 80 = 4. Image y increases downward. Both axes use the same scale, so the vector really is three times longer horizontally than vertically.
3. Assume Similar Brightness
Classical methods assume a tracked point keeps approximately the same brightness. But many pixels can share intensity 145. A single matching value is not enough; methods use local patterns and additional assumptions to constrain the displacement.
4. Read A Dense Field
This display samples a dense field so individual arrows remain visible. In the fixed-camera example, the moving patch has displacement (12,4); static background points have zero flow. With camera motion, the background can have nonzero flow too.
5. Combine Local Constraints
Linearizing brightness constancy gives Ix u + Iy v + It = 0: spatial gradients, motion components, and temporal change. In this small-motion toy example, two samples imply u=2, v=1. Lucas–Kanade combines neighborhood constraints with a least-squares fit under shared local motion.
6. Use An Image Pyramid
A 12-pixel shift becomes 6 at half resolution and 3 at quarter resolution. Estimate at a coarse scale, enlarge the estimate, then refine. A larger support window is a different choice: it adds evidence but may mix motions across object boundaries.
7. Understand The Aperture Problem
The three arrows share the same horizontal component. An untextured vertical edge cannot reveal their different vertical components within this window. The observed boundary moves identically in every case. This is ambiguity in the evidence, not a numerical solver error.
8. Prefer Informative Patches
A corner adds changes in a second direction, helping constrain both components. A flat patch supplies little information in either direction. Larger neighborhoods can help, provided they contain useful texture and approximately share one motion.
9. Handle Occlusion
When a point moves behind an obstacle, its correspondence is unobserved. A method may infer its motion using neighboring evidence, but that differs from matching a visible point. Confidence checks and occlusion handling matter when consuming a flow estimate.
10. Where It Breaks
The patch stays still while its intensities increase by a factor of 1.5. Raw brightness matching no longer fits the true zero displacement. Illumination changes, reflections, blur, and scene changes require more robust evidence; the constancy assumption is an approximation.
The Quick Version
- Optical flow describes apparent movement in image coordinates.
- Displacement is the difference between matched pixel positions.
- Equal brightness is useful evidence, not a unique match.
- Dense flow assigns vectors everywhere; displays may sample them.
- Independent local gradients help determine both motion components.
- Pyramids reduce displacement measured at coarse image scales.
- A straight edge hides motion parallel to itself.
- Corners constrain more directions than flat or straight-edge patches.
- Occluded points have no visible next-frame correspondence.
- Lighting changes can violate brightness constancy without any motion.