Triangulation from Two Image Rays

Each camera turns a pixel into a ray in space. Triangulation is where those two rays come closest, and that closest point is the 3D answer.

Two back-projected rays meet at the triangulated point with a small reprojection residual

Why Does This Exist?

Matches alone are flat. Given a left pixel, a right pixel, and known cameras, something must lift the pair back into 3D, and that lifter is triangulation. Every structure from motion pipeline, every stereo vision depth map, and every SLAM map point passes through it. Without it there are correspondences but no shape.

This page covers the two-view lift and its residual. The match constraint lives in the fundamental matrix; the joint refinement over many points lives in bundle adjustment.

Think of It Like This

Two laser pointers crossing in fog

Two friends shine laser pointers at the same balloon in fog. Each beam alone says direction only; where the beams cross says distance. When the fog jitters, the beams miss by a hair and the crossing becomes the shortest bridge between them.

Pixels are laser directions and triangulation is that bridge midpoint. The analogy stops at calibration: friends aim freely while cameras need known poses before any beam means anything.

How It Actually Works

A calibrated camera writes $x = P X$ with $P = K [R | t]$ , where $x$ is the homogeneous pixel and $X$ the homogeneous 3D point. The direct linear transform builds two rows per view from the cross product $x \times (P X) = 0$ and solves the stacked $4 \times 4$ system by SVD, taking the smallest singular vector as $X$ . In noise the rays skew, so the midpoint of the common perpendicular replaces the exact crossing.

Quality is reported as reprojection residual: project $X$ back into each view and measure pixel distance $\|x - \hat{x}\|$ . Then comes the cheirality check: keep $X$ only when its camera-frame depth $Z$ is positive in both views.

Worked numbers

Use $f = 800$ , center $320$ , left camera at the origin and right camera at $x = 0.2$ m. Left pixel $u_1 = 480$ gives ray slope $(480 - 320) / 800 = 0.2$ ; right pixel $u_2 = 400$ from $x = 0.2$ gives $(400 - 320) / 800 = 0.1$ relative to its own center. Solving $Z \cdot 0.2 = 0.2 + Z \cdot 0.1$ gives $Z \cdot 0.1 = 0.2$ , so $Z = 2.0$ m and $X = 0.4$ m. Reprojecting $(0.4, 0, 2.0)$ returns exactly $(480, 400)$ , residual $0.0$ px; shifting $u_2$ to $401$ moves the residual to about $1.0$ px and depth to $1.98$ m.

Code

1def triangulate_depth(u1: float, u2: float, f: float, cx: float, baseline: float) -> float:2    slope1 = (u1 - cx) / f  # left ray direction X / Z3    slope2 = (u2 - cx) / f  # right ray direction relative to its center4    return baseline / (slope1 - slope2)  # depths where the rays meet5
6
7print(triangulate_depth(480.0, 400.0, 800.0, 320.0, 0.2))  # -> 2.0

Watch Out For

Triangulating points at tiny parallax

When the two rays are nearly parallel, a one-pixel shift moves depth by meters. The symptom is far points with huge variance that still report sub-pixel residuals. Reject points whose parallax angle sits under about one degree, or down-weight them heavily in the next optimization.

Skipping the cheirality test

Algebra returns an $X$ even when it sits behind a camera, and its reprojection error can look excellent. Downstream code then fuses ghost geometry. Always transform into each camera frame and require $Z > 0$ twice.

The Quick Version

Triangulation lifts one match plus two known cameras into one 3D point.
The linear method stacks $x \times (P X) = 0$ rows and solves by SVD.
Real rays skew, so the answer is the midpoint of their common perpendicular.
Reprojection residual in pixels is the quality number to watch.
Pixels $480$ and $400$ with $f$ $800$ and baseline $0.2$ m meet at $2.0$ m.
Tiny parallax and negative depth are the two traps to filter.