Multi-View Stereo for Dense Depth

One pair gives a noisy shell; many pairs fill the holes. Multi-view stereo fuses depth from several viewpoints into one dense cloud.

Depth planes sweep through space and photo-consistent depths survive the vote

Why Does This Exist?

Structure from motion returns sparse points and camera poses, which is enough to navigate but not to render or grasp. Multi-view stereo exists to densify that skeleton: using the known poses, it estimates a depth for nearly every pixel by asking which depth makes all views agree. It is the step behind COLMAP dense clouds and offline 3D scans.

This page covers the sweep-and-fuse mechanism. Sparse poses come from structure from motion; joint pose-point polish is bundle adjustment. Single-pair depth is stereo vision.

Think of It Like This

Flipping through frosted glass sheets

Hold a photo up behind a stack of frosted glass sheets and slide it back until the view through the stack lines up with neighboring photos. Only one depth lines up everywhere; sheets in front or behind look smeared.

Depth planes are the glass sheets and photo agreement is the sharpness test. The analogy stops at occlusion: glass never blocks while real surfaces hide, so votes from hidden views must be dropped.

How It Actually Works

For a reference view, the method sweeps candidate depths $d$ along its ray and projects each hypothesis into $N$ neighbor views. Photometric consistency scores each depth by patch similarity across views, often normalized cross-correlation over $5 \times 5$ windows. The cost volume $C(u, v, d)$ holds those scores; the depth with minimum cost wins per pixel, then geometric consistency filters keep only depths that reproject within about one pixel in at least three views. Surviving depths fuse into one cloud through weighted averaging by view angle and baseline.

Worked numbers

Candidate depths $1.0$ , $1.2$ , $1.4$ m score mean patch errors $8.2$ , $0.5$ , $6.9$ across five views. The winner $1.2$ m has variance $0.5$ px versus $2.1$ px for the single best pair, so fusion cuts error roughly fourfold. Filtering drops $18$ percent of pixels that agree in two views but miss the third by over $1.0$ px, mostly at occlusions.

Code

1def fuse_depth(costs: list[float], depths: list[float]) -> float:2    best = min(range(len(costs)), key=lambda i: costs[i])  # lowest photo error wins3    return depths[best]4
5
6print(fuse_depth([8.2, 0.5, 6.9], [1.0, 1.2, 1.4]))  # -> 1.2

Watch Out For

Fusing textureless and reflective areas

Blank walls give flat cost volumes where every depth scores the same, and mirrors give sharp but wrong minima from reflections. The symptom is lumpy walls and doubled shiny objects. Mask low-texture and specular pixels before fusion instead of averaging them in.

The Quick Version

Multi-view stereo densifies sparse poses into per-pixel depth.
Plane sweep builds a cost volume of photo agreement across neighbor views.
Minimum-cost depth wins, then geometric checks across three views filter.
Surviving depths fuse by baseline-weighted averaging into one cloud.
Five agreeing views beat one pair by roughly fourfold in variance.
Textureless and reflective pixels must be masked, not fused.