Depth Completion from Sparse Lidar

Lidar gives a sprinkle of true depths on the photo. Depth completion spreads those seeds along image edges into a full metric map.

Sparse lidar seeds grow along image edges into dense metric depth

Why Does This Exist?

Lidar returns exact but sparse points, about $5$ percent of pixels on driving benchmarks, while planning wants every pixel. Depth completion exists to fill the rest without losing metric truth: it propagates seed depths along image structure instead of hallucinating from RGB alone. Pure monocular depth estimation cannot promise scale; completion anchors every region to nearby seeds.

Overview is depth estimation; single-image priors are monocular depth; stereo metric depth is stereo depth estimation.

Think of It Like This

Rain gauges plus cloud photos

A few rain gauges report exact rainfall while satellite clouds show where rain falls. Interpolating gauges alone smears across dry ridges; guiding by cloud edges keeps wet and dry apart.

Gauges are lidar seeds and clouds are RGB edges. The analogy stops at motion: clouds drift while image edges sit still during the fill.

How It Actually Works

The net takes RGB plus sparse depth and a validity mask. Sparsity-aware convolutions normalize by valid count so empty pixels contribute zero, not black depth. Early fusion concatenates seeds with image features; spatial propagation then iterates $D^{k+1} = A D^k$ with affinities $A$ learned from RGB, spreading depth along similar surfaces but not across edges. Loss scores RMSE only on valid ground-truth pixels, keeping empty areas supervised by propagation rather than direct targets.

Worked numbers

A $1000 \times 300$ frame holds $300000$ pixels with $15000$ seeds ( $5$ percent). Nearest-neighbor fill gives RMSE $1.6$ m; guided propagation reaches $0.8$ m by stopping at car edges. Each propagation step spreads about $3$ px, so $20$ steps bridge the typical $60$ px gap between scan lines.

Code

1def valid_mean(values: list[float], mask: list[int]) -> float:2    valid = [v for v, m in zip(values, mask) if m]  # ignore empty pixels3    return sum(valid) / len(valid)4
5
6print(valid_mean([3.0, 0.0, 5.0], [1, 0, 1]))  # -> 4.0

Watch Out For

Smearing depth across occlusion edges

Weak affinities leak foreground depth onto background through thin poles and signs. The symptom is halos of near depth around every edge. Sharpen affinity with edge supervision and stop propagation after a fixed radius instead of iterating to convergence.

The Quick Version

Depth completion densifies sparse metric seeds guided by the image.
Validity masks keep empty pixels from poisoning convolutions.
Learned affinities spread depth along surfaces, not across edges.
Five percent seeds can reach RMSE near $0.8$ m on driving data.
Score only valid ground-truth pixels during training.
Halos at edges signal over-propagation to fix.