Epipolar Geometry & Stereo Matching

How two cameras see the same 3D point in the world and use their geometry to simplify finding matches.

Epipolar lines restrict the search space for stereo matching from 2D to 1D.

Why Does This Exist?

When capturing a 3D scene with two cameras (stereo vision), finding which pixel in the left image corresponds to the same physical point in the right image is a massive computational problem. If you simply search everywhere, it takes too long and produces many false matches. Epipolar geometry exists to mathematically constrain this search space. By understanding the geometric relationship between the two cameras, we can reduce the search from a full 2D image down to a single 1D line (the epipolar line), making stereo matching fast and reliable.

Think of It Like This

Searching across two parallel strings

Imagine two people standing apart, both holding a string attached to a single kite in the sky. If the person on the left wants to know exactly where the person on the right is looking to see the kite, they don't need to search the entire sky. They just need to look along the string the other person is holding.

Epipolar geometry works exactly like this. The "string" connecting the right camera to the 3D point projects as a straight line (the epipolar line) onto the left camera's sensor. The left camera only needs to search along that line to find the match.

How It Actually Works

The core of epipolar geometry relies on a few key concepts:

Baseline: The line connecting the optical centers of the two cameras.
Epipoles: The points where the baseline intersects the image planes (essentially where each camera sees the other camera).
Epipolar Plane: The plane formed by the 3D point and the two camera centers.
Epipolar Line: The intersection of the epipolar plane with the image plane.

When a point $X$ in 3D space is observed as $x$ in the left camera, the corresponding point $x'$ in the right camera must lie on the epipolar line corresponding to $x$ . This constraint is mathematically represented by the Fundamental Matrix ( $F$ ) for uncalibrated cameras, or the Essential Matrix ( $E$ ) for calibrated cameras.

The equation is: $x'^T F x = 0$

Stereo Matching then uses this constraint. Instead of searching a whole 2D image for a patch that looks like $x$ , algorithms only search along the 1D epipolar line. Once a match is found, the difference in their horizontal coordinates (after rectification) is the disparity. Depth is inversely proportional to disparity: closer objects have larger disparity.

Code

1import cv22import numpy as np3
4# Assuming points1 and points2 are matching points found via SIFT/ORB5# Compute the Fundamental Matrix6F, mask = cv2.findFundamentalMat(points1, points2, cv2.FM_RANSAC)7
8# Given a point in image 1, find the epipolar line in image 29# lines contains the coefficients (a, b, c) for the line equation ax + by + c = 010lines2 = cv2.computeCorrespondEpilines(points1.reshape(-1, 1, 2), 1, F)11lines2 = lines2.reshape(-1, 3)12
13# Search along this line for stereo matching (simplified)

Watch Out For

Textureless Regions

Stereo matching relies on finding visually similar patches. If the epipolar line passes through a completely flat, textureless wall, the algorithm won't be able to confidently find the matching point, leading to holes in the depth map.

Occlusions

A 3D point visible in the left camera might be hidden behind another object in the right camera. In this case, searching the epipolar line will yield an incorrect match or no match at all.

The Quick Version

Epipolar geometry defines the mathematical relationship between two camera views of the same scene.
It turns the 2D search for matching pixels into a 1D search along epipolar lines.
The Fundamental Matrix algebraically represents this geometry.
Stereo matching uses these lines to efficiently find corresponding points and compute depth (disparity).