Stereo Vision and Depth from Disparity

Observe One Point Twice

Stereo estimates depth by finding the same scene point in two separated views and intersecting its viewing rays. The distance between camera centers is the baseline. Once camera geometry is known, the point's horizontal image displacement can determine metric depth. The triangle here is qualitative; later beats supply scales and numbers. Correct correspondence is essential: intersecting rays from different scene points gives a meaningless reconstruction. Epipolar geometry explains the search constraint before rectification.

Align Corresponding Image Rows

The upper pair shows corresponding epipolar lines with different slopes. Rectification resamples the views so those lines become matching horizontal rows in the lower pair. Search can then compare x positions at the same rectified y coordinate. This is a schematic change of image coordinates, not a physical rotation of the scene or a numerical calibration example. Rectification alone does not select the correct point along a row.

Use consistent rectified camera matrices and handle lens distortion first or in the combined mapping. The OpenCV calibration module defines stereo rectification and its reprojection matrix. Camera calibration establishes the intrinsics and distortion model.

Measure Horizontal Disparity

Define disparity as d=uL−uR. A left pixel at 336 and its right partner at 288 give d=48 pixels. Both strips share the same cropped horizontal coordinates, drawn at two units per pixel, so the bracket length is 96. The strips are separated vertically to compare them; their actual rectified row coordinate is the same. Disparity is not a depth in meters.

Our fixture assumes parallel rectified axes, common focal length 800 pixels and aligned principal points (320,240). If rectification produces different principal points, the offset must be included; raw uL−uR alone is no longer the simplified denominator used below.

Triangulate Optical Depth

Place the left center at X=0, the right at X=B=0.12 meters. With common focal length f and aligned principal points, their centered horizontal projections are fX/Z and f(X−B)/Z. Subtracting cancels X, leaving d=fB/Z, hence:

$$Z=\frac{fB}{d}=\frac{800\ \text{px}\times0.12\ \text{m}}{48\ \text{px}}=2\ \text{m}.$$

The same point has X=(336−320)×2/800=0.04 meters. Its normalized left ray is 0.02 and right ray (0.04−0.12)/2=−0.04; they intersect at (X,Z)=(0.04,2). The top view uses 1,000 drawing units per meter horizontally and 90 vertically to expose the small baseline. Angles on that anisotropic drawing are not physical ray angles. Z is optical-axis depth, not Euclidean range. The official depth derivation uses the same similar-triangle relation.

Read Inverse Depth

Keep fB=96 pixel-meters. Depths 2, 4 and 12 meters produce disparities 96/2=48, 96/4=24 and 96/12=8 pixels. Doubling depth halves disparity. The horizontal depth axis uses 35 drawing units per meter; stems use four per disparity pixel. The large gap between 4 and 12 is intentional. A disparity map's values are therefore proportional to inverse depth, so displaying it as if it were linear metric distance changes the interpretation.

Compare Candidate Patch Costs

For a toy left patch [10,20,30], compare three right patches along its row. At candidate disparities 47, 48 and 49, use [20,30,40], [9,21,29] and [21,29,39]. Their sums of absolute differences are 10+10+10=30, 1+1+1=3, and 11+9+9=29. The center candidate wins. Bars share six drawing units per cost unit.

This is a synthetic three-sample SAD example, not a complete stereo implementation. Real methods aggregate evidence spatially, penalize implausible disparity changes and reject ambiguous minima. Flat or repeated texture can give multiple similar costs; a larger window can mix foreground and background depths. OpenCV's StereoSGBM reference describes its different cost construction and regularization.

Refine Between Integer Samples

Switch to a smoother toy cost neighborhood [5,2,3] at disparities 47, 48 and 49. With t=d−48, the quadratic C(t)=2t²−t+2 passes through all three samples. Its derivative 4t−1 vanishes at t=0.25, giving disparity 48.25 and minimum cost 1.875. Equivalently, the local offset is 0.5×(5−3)/(5−2×2+3)=0.25.

The curve uses 160 drawing units per disparity pixel and 35 per cost unit. Subpixel fitting refines the location of a well-behaved minimum; it does not create independent image information. A flat, noisy or multi-peaked neighborhood may not support that fit. Real matcher arithmetic can quantize results. For OpenCV BM/SGBM fixed-point outputs, convert the stored values to pixel disparities using the documented scale before reprojection; commonly that means division by 16. Floating-point outputs require their own contract.

Check The Reverse Match

The forward match sends uL=336 to uR=336−48=288. Suppose a right-to-left check estimates a positive reverse displacement magnitude 44. It returns 288+44=332, four pixels from the starting location, failing an illustrative one-pixel tolerance. The crop uses four drawing units per coordinate pixel; the amber cross marks 332 rather than 336. Its routed dashed loop represents the reverse lookup, not motion through space.

Some implementations store reverse disparities with the opposite sign, so adapt the arithmetic to their convention. Occluded pixels may have no valid partner; repeated patterns can produce inconsistent or even consistently wrong matches. A consistency check marks unreliable evidence, not a replacement depth estimate.

Choose A Useful Baseline

At depth 4 meters with f 800 pixels, baseline 0.06 meters gives d=12 pixels. Doubling B to 0.12 gives 24. Both top views place the point at X=0.04,Z=4; they use 1,000 drawing units per meter in X and 40 in Z. The lower disparity brackets use a separate four-units-per-pixel scale, so their lengths are 48 and 96.

A wider baseline gives a larger measurable displacement at the same range, but also larger viewpoint differences, occluded regions and potentially less overlap. It must suit the intended working distance and matching search range. It is not a free increase in usable depth quality.

Where It Breaks

At disparity 8 pixels, Z=96/8=12 meters. A one-pixel error changes that to 96/9≈10.667 or 96/7≈13.714 meters. The metric axis uses 80 drawing units per meter; its amber interval is asymmetric about the white 12-meter estimate because depth is reciprocal. This interval propagates a bounded disparity perturbation; it is not a measured statistical confidence interval.

Local sensitivity is |dZ/dd|=fB/d². At d 8, a half-pixel perturbation gives a first-order depth change of 96/64×0.5=0.75 meters. As disparity approaches zero, metric depth becomes ill-conditioned. A zero or invalid matcher output must not be accepted as reliable infinite depth. Calibration error, motion between unsynchronized images, reflective surfaces and poor correspondence can dominate this ideal uncertainty calculation. Keep invalid regions masked and distinguish missing evidence from a far surface.

The Quick Version

• Separated cameras observe each shared point along two rays.
• Rectification aligns corresponding epipolar lines with image rows.
• Disparity subtracts corresponding horizontal pixel coordinates.
• Calibrated focal length and baseline turn disparity into metric depth.
• Disparity is proportional to inverse optical depth.
• Matching costs compare candidate appearances along a rectified row.
• Subpixel interpolation refines a supported local cost minimum.
• Reverse checks reject inconsistent correspondences.
• Wider baselines increase disparity while complicating correspondence.
• Small disparity magnifies depth error and demands validity checks.