Scene Flow Estimation in 3D Motion

Give every 3D point its own velocity arrow. Scene flow lifts frame-to-frame matching plus depth change into true 3D motion.

Two point clouds link by motion arrows with depth change setting length

Why Does This Exist?

Two-frame 2D flow cannot warn which points approach. Scene flow exists for that 3D answer in driving and manipulation: it outputs per-point $(V_x, V_y, V_z)$ so planners brake for approaching bumpers rather than large but distant trucks. The overview is optical flow and scene flow; 2D baselines are RAFT.

This page covers the 3D lift across stereo, RGB-D, and lidar variants.

Think of It Like This

Tagging birds between two photos

Two flock photos a second apart show shifted dots. Match each bird, then note which ones also grew larger: those approach. Flat shift plus size change gives the full flight arrow.

Matches are flow and size change is depth rate. The analogy stops at occlusions: birds rarely hide while bumpers occlude constantly, so half the arrows need outlier flags.

How It Actually Works

Stereo scene flow jointly solves optical flow $(u, v)$ plus disparity $d$ and disparity change $\Delta d$ , converting to 3D through $Z = f B / d$ and differencing across frames. RGB-D variants read $Z$ directly and solve a rigid-plus-nonrigid warp with point-to-plane residuals. Lidar scene flow registers clouds with coarse matches plus refined flow, supervised by chamfer distance plus cycle consistency when labels are missing. Outlier rates near $15$ percent on occlusions are standard and must be masked.

Worked numbers

Stereo with $f = 600$ px, $B = 0.5$ m, $Z = 6.0$ m, flow $(5, 0)$ px, disparity change from $50.0$ to $52.6$ px gives $\Delta Z \approx -0.3$ m. The 3D vector is $(0.05, 0.0, -0.3)$ m per frame. 3D endpoint error against truth $(0.06, 0.0, -0.28)$ is $\sqrt{0.01^2 + 0.02^2} \approx 0.022$ m.

Code

1def flow_3d_error(pred: tuple[float, float, float], gt: tuple[float, float, float]) -> float:2    return sum((p - g) ** 2 for p, g in zip(pred, gt)) ** 0.5  # meters3
4
5print(round(flow_3d_error((0.05, 0.0, -0.3), (0.06, 0.0, -0.28)), 3))  # -> 0.022

Watch Out For

Averaging flow across motion boundaries

Wheels, bumpers, and backgrounds share pixels at edges; naive smoothing blends their vectors. The symptom is smeared arrows that miss both motions. Edge-aware regularization or motion segmentation must precede smoothing.

The Quick Version

Scene flow outputs a 3D velocity per point, not a 2D shift.
Stereo, RGB-D, and lidar variants differ only in the depth source.
Joint flow plus disparity change gives metric 3D directly.
Reference case is $0.05$ m lateral with $0.3$ m approach at $6$ m.
3D endpoint error in meters is the score to report.
Occlusions near $15$ percent must be masked, not averaged.