Epipolar Geometry

Epipolar geometry connects two views of the same 3D scene, showing how a point in one image restricts where it can appear in the other.

$One left pixel defines one right epipolar line, and the true match zeroes the epipolar constraint$

One left pixel defines one right epipolar line, and the true match zeroes the epipolar constraint

Why Does This Exist?

When two cameras look at the same 3D scene, finding which pixel in one image corresponds to a pixel in the other is a massive search problem. Epipolar geometry restricts this search to a single line instead of a whole 2D image, making depth estimation computationally feasible.

Conventions used here, shared with the vision side: homogeneous pixels $x = (u, v, 1)$ , cameras $P = K[R|t]$ with intrinsics $K$ , image $y$ pointing down after rectification, and positive depth $Z > 0$ in front of each camera. The calibrated story continues in the essential matrix, the uncalibrated matrix in the fundamental matrix, and the dense application in stereo vision. This page is the canonical math owner; the applied tour lives in epipolar geometry and stereo matching.

The one idea you need first

A homogeneous pixel $x = (u, v, 1)$ is a sight direction, not just a location. Scaling it does not change what it means, which is why matrices can act on it with straight multiplication. The full treatment is in homogeneous coordinates and projective geometry.

Think of It Like This

Two witnesses sharing one sight-line

Imagine two people looking at a distant mountain peak from different positions. If you draw a line from each person's eye to the peak, these lines and the line connecting the two people form a geometric plane. One witness can draw the other a single sight-line across their view: the peak must sit somewhere along it.

Pixels are sight directions and the epipolar line is the drawn sight-line. The analogy stops at noise: witnesses point exactly while pixels quantize, so the line test becomes a small residual rather than exact zero.

How It Actually Works

Epipolar geometry relies on a few fundamental concepts:

Epipoles: The point where the baseline (the line connecting the two camera centers) intersects each image plane.
Epipolar Plane: The plane formed by the 3D point and the two camera centers.
Epipolar Lines: The intersection of the epipolar plane with each image plane. If a point is seen at a specific pixel in one camera, it must lie on the corresponding epipolar line in the other camera.
Fundamental Matrix: A $3 \times 3$ rank-2 matrix $F$ for uncalibrated pairs with $x'^T F x = 0$ .
Essential Matrix: A $3 \times 3$ matrix $E = [t]_{\times} R$ for calibrated pairs with $\hat{x}'^T E \hat{x} = 0$ on normalized coordinates $\hat{x} = K^{-1} x$ . It carries five degrees of freedom: three for rotation plus two for translation direction, since scale is unobservable.

Scale ambiguity is fundamental: $E$ and $F$ fix translation direction but never its length, so two-view reconstruction is metric only up to one global scale. Cheirality then selects the valid solution among the four $R$ and $t$ decompositions by requiring triangulated points to sit with $Z > 0$ in both cameras.

Worked example

Pure $x$ translation gives $F$ with rows $(0,0,0)$ , $(0,0,-1)$ , $(0,1,0)$ . Take $x = (2, 1, 1)$ :

l' = F x = (0,\ -1,\ 1)

so the epipolar line is $v = 1$ . The true match $x' = (5, 1, 1)$ scores $5 \times 0 + 1 \times (-1) + 1 \times 1 = 0$ . A wrong match $(5, 2, 1)$ scores $-1$ and is rejected.

The same geometry gives depth from disparity. With $f = 800$ px and baseline $0.12$ m at disparity $d = 80$ px, depth is $Z = 800 \times 0.12 / 80 = 1.2$ m. Lifting a matched pair back to 3D is triangulation.

Code

1def epipolar_residual(xp: tuple[float, float, float], x: tuple[float, float, float]) -> float:2    f = ((0.0, 0.0, 0.0), (0.0, 0.0, -1.0), (0.0, 1.0, 0.0))  # pure x translation3    lx = [sum(f[r][c] * x[c] for c in range(3)) for r in range(3)]  # l' = F x4    return sum(xp[r] * lx[r] for r in range(3))  # x' dot l', zero at a true match5
6
7print(epipolar_residual((5.0, 1.0, 1.0), (2.0, 1.0, 1.0)))  # -> 0.08print(epipolar_residual((5.0, 2.0, 1.0), (2.0, 1.0, 1.0)))  # -> -1.0

Watch Out For

Estimating geometry from flat or rotation-only motion

If every matched point lies on one plane, a homography explains all matches and $F$ turns degenerate with fiction lines. Pure rotation gives no baseline at all, so depth is unobservable and triangulation returns noise. The symptom is low RANSAC error with absurd structure. Test the homography first on flat scenes and require translation parallax before triangulating.

Trusting textureless stereo along the line

The epipolar line says where to look, not what matches: on blank walls every candidate scores the same and disparity noise explodes into depth. Far range squares the damage through $Z^2 / (f B)$ . Mask low-texture rows and clamp usable range by baseline instead of accepting every argmin.

The Quick Version

Epipolar geometry cuts two-view matching from a 2D search to a 1D line search.
The uncalibrated constraint is $x'^T F x = 0$ with rank-2 $F$ ; calibrated is $\hat{x}'^T E \hat{x} = 0$ .
The essential matrix factors as $E = [t]_{\times} R$ with five degrees of freedom.
Translation scale is unobservable from two views alone; cheirality picks the front-facing solution.
True matches score zero; $(5, 1)$ passes while $(5, 2)$ scores $-1$ .
Depth follows $Z = f B / d$ ; $800 \times 0.12 / 80 = 1.2$ m.
Flat scenes, pure rotation, and textureless rows are the three standard degeneracies.