Fundamental Matrix of Two Camera Views

The fundamental matrix is the 3 by 3 table that encodes how two uncalibrated cameras sit relative to each other. It turns a pixel in one image into a search line in the other, and the true match always scores zero.

F turns a left pixel into a right epipolar line; the true match makes x transpose F x equal zero

Why Does This Exist?

Matching a pixel across two views by searching the whole second image is slow and ambiguous. Epipolar geometry cuts the search to one line, and the fundamental matrix $F$ is that line in algebraic form, needing no calibration data at all. It is the workhorse behind uncalibrated reconstruction: estimate $F$ from matches, then recover cameras and structure. The stereo-flavored tour is epipolar geometry and stereo matching; this page is the matrix itself. Its calibrated sibling is the essential matrix.

Think of It Like This

Two witnesses, one lineup

Witness A says the suspect stood third from the left. Witness B, standing elsewhere, cannot reuse "third from the left," but A can draw B a sight-line across the lineup room: the suspect must stand somewhere along that line. B still has to look, yet the search collapses from a room to a line.

$F$ is the written sight-line rule between two cameras. The analogy stops at recognition: the line says where to look, while appearance matching along it remains its own problem.

How It Actually Works

For corresponding homogeneous pixels $x$ and $x'$ , the epipolar constraint reads $x'^T F x = 0$ . Here $F$ is $3 \times 3$ with rank $2$ : its left and right null spaces are the two epipoles, which is why every epipolar line passes through its epipole. Counting freedoms gives $9 - 1$ (global scale) $- 1$ (rank constraint) $= 7$ degrees of freedom, so seven point pairs determine $F$ in principle.

In practice the $8$ -point algorithm stacks one linear row per match and solves by SVD, then projects the answer onto rank $2$ by zeroing its smallest singular value. RANSAC wraps the solver to survive mismatches. With intrinsics $K$ and $K'$ known, $F$ upgrades to $E = K'^T F K$ , the subject of the next page.

Worked numbers

Pure translation along the $x$ axis gives $F$ with rows $(0,0,0)$ , $(0,0,-1)$ , $(0,1,0)$ . Take $x = (2, 1, 1)$ :

l' = F x = (0,\ -1,\ 1)

so the epipolar line is $0 \cdot u - 1 \cdot v + 1 = 0$ , the horizontal line $v = 1$ . The true match $x' = (5, 1, 1)$ lies on it, and the constraint confirms:

x'^T F x = 5 \times 0 + 1 \times (-1) + 1 \times 1 = 0

A wrong match such as $(5, 2, 1)$ would score $-1$ , nonzero and therefore rejected.

Code

1def epipolar_residual(xp: tuple[float, float, float], x: tuple[float, float, float]) -> float:2    f = ((0.0, 0.0, 0.0), (0.0, 0.0, -1.0), (0.0, 1.0, 0.0))  # pure x translation3    lx = [sum(f[r][c] * x[c] for c in range(3)) for r in range(3)]  # l' = F x4    return sum(xp[r] * lx[r] for r in range(3))  # x' dot l', zero at a true match5
6
7print(epipolar_residual((5.0, 1.0, 1.0), (2.0, 1.0, 1.0)))  # -> 0.0

Watch Out For

Estimating F from a flat scene

If every matched point lies on one plane, a homography explains all matches and $F$ becomes degenerate: the solver returns confidently with tiny residuals while the epipolar lines are fiction. Symptoms are plausible RANSAC scores with absurd geometry. Test a homography first on suspiciously flat scenes, and estimate the plane map instead, as homography estimation does.

Skipping the rank-2 projection

The raw $8$ -point answer is full rank because of noise, but a true fundamental matrix has a zero singular value. Without zeroing the smallest one, epipolar lines systematically miss their matches by a few pixels and downstream triangulation inherits the bias. One SVD truncation fixes it: solve, decompose, zero, rebuild.

The Quick Version

The fundamental matrix encodes uncalibrated two-view geometry in one $3 \times 3$ rank-2 matrix.
The epipolar constraint $x'^T F x = 0$ tests any candidate match algebraically.
$F$ has $7$ degrees of freedom: nine entries minus scale minus the rank constraint.
The $8$ -point algorithm plus SVD rank projection is the standard estimator.
With known intrinsics, $F$ upgrades to the essential matrix via $E = K'^T F K$ .
Flat scenes make $F$ degenerate; a homography owns that case instead.