Optical Flow plus Scene Flow Guide

Track where each pixel went, then lift that motion into 3D. Optical flow is the 2D shift; scene flow adds depth change to make it 3D.

Pixel shifts plus depth change combine into 3D scene motion vectors

Why Does This Exist?

Optical flow tells a car that pixels moved, but not whether the object approaches or just slides sideways. Scene flow exists to answer in meters: combine 2D flow with depth change from stereo, RGB-D, or lidar into a 3D vector per point. This overview separates the two so later pages can specialize.

Classical 2D methods start at optical flow; learned 2D models are FlowNet, PWC-Net, RAFT, and GMFlow. The 3D page is scene flow estimation.

Think of It Like This

Shadows versus real steps

Watch a puppet shadow slide across a wall: direction is clear but distance is not. Step into the room and see the puppet stride forward half a meter while sliding sideways: that full stride is the truth.

Shadows are optical flow and strides are scene flow. The analogy stops at density: shadows cover walls while scene flow needs depth everywhere, which is often missing.

How It Actually Works

Optical flow $(u, v)$ obeys brightness constancy $I_x u + I_y v + I_t = 0$ with one equation for two unknowns, resolved by local windows or global smoothness. Scene flow adds depth rate: with depth $Z$ and intrinsics $f$ , the 3D velocity is $V = (Z u / f, Z v / f, \dot{Z})$ . Stereo scene flow solves flow plus disparity change together; RGB-D reads $\dot{Z}$ from the sensor; lidar scene flow registers clouds. Endpoint error $\sqrt{(u - \hat{u})^2 + (v - \hat{v})^2}$ scores 2D; 3D error adds $|\dot{Z} - \hat{\dot{Z}}|$ .

Worked numbers

A pixel shifts $(u, v) = (5, 0)$ px with $Z = 6.0$ m, $f = 600$ px, and depth change $\dot{Z} = -0.3$ m per frame (approaching). Then $V_x = 6.0 \times 5 / 600 = 0.05$ m, $V_y = 0$ , $V_z = -0.3$ m. Endpoint error against prediction $(4, 1)$ is $\sqrt{1 + 1} \approx 1.41$ px.

Code

1def scene_velocity(u: float, v: float, z: float, f: float, dz: float) -> tuple[float, float, float]:2    return (z * u / f, z * v / f, dz)  # lift 2D flow plus depth rate to 3D3
4
5print(scene_velocity(5.0, 0.0, 6.0, 600.0, -0.3))  # -> (0.05, 0.0, -0.3)

Watch Out For

Reading flow as 3D speed

Large 2D flow can mean a near object creeping or a far object racing; without depth the two are identical. The symptom is emergency braking for distant trucks. Always multiply by depth over focal length before acting on flow magnitude.

The Quick Version

Optical flow is 2D pixel shift $(u, v)$ between frames.
Scene flow is 3D velocity $(V_x, V_y, V_z)$ in meters.
Depth plus flow lifts 2D into 3D through $Z u / f$ scaling.
Endpoint error scores 2D; add depth-rate error for 3D.
A $5$ px shift at $6$ m with $0.3$ m approach is the reference case.
Flow without depth cannot separate near-slow from far-fast.