Structure from Motion

One Scene, Several Views

Structure from Motion (SfM) estimates camera positions and sparse 3D points from overlapping photos. It seeks one scene that explains observations across images, rather than rebuilding each image independently. The rays meet at the same outlined point; the drawing is a spatial slice, not a timeline.

The overview expands horizontal camera separation for visibility. C1 and C2 retain the camera identities used in the worked pair below; the triangulation detail uses equal spatial scales on both axes.

COLMAP's tutorial separates sparse SfM from the later dense multi-view stereo stage. A sparse model contains reconstructed feature locations, not a complete surface. The scene must remain sufficiently rigid across views.

Connect Feature Tracks

The connected circles are observations A1, B2 and C1 of one proposed feature. The squares are other features. Joining pairwise matches into a track lets several cameras constrain one 3D point. The connecting line represents identity across images, not a physical path through space.

Appearance similarity starts the search; matching and RANSAC explain why it needs geometric verification. Repeated windows can create plausible but incorrect tracks.

Check Image Geometry

For this parallel, rectified pair, corresponding observations lie on the same row. The cross is inconsistent with that constraint. General SfM images need not already be rectified: an estimated fundamental matrix predicts an epipolar line, which can be slanted. A point on the line remains only a candidate.

With known intrinsics, normalized coordinates allow an essential matrix to constrain relative rotation and translation direction. Robust estimation must tolerate incorrect matches; calibration and geometry are evidence, not substitutes for correspondence.

Choose A Physical Pose

Relative-pose decomposition supplies multiple mathematical candidates. Cheirality tests whether triangulated points have positive camera-frame depth. This paired slice isolates two depth-sign alternatives: forward is upward, with depths +4 and −4 drawn on the same scale. The backward intersection must be rejected.

The picture illustrates one rejection test, not the full essential-matrix decomposition. A real initializer evaluates the pose candidates over many correspondences and needs enough parallax.

Intersect The Rays

Use calibrated parallel cameras at X=0 and X=1, focal length 100 pixels, and point (X,Z)=(0.5,4). Centered horizontal image coordinates are u1=100×0.5/4=12.5 and u2=100×(0.5−1)/4=−12.5. Disparity is 25 pixels, so Z=fB/d=100×1/25=4.

The drawing uses the same 50 units per spatial unit on both axes. This ideal pair has exact ray intersection; noisy multi-view triangulation instead fits imperfect observations. Stereo depth develops the disparity relationship.

Notice The Missing Scale

Both triangles use a common spatial display scale. Doubling baseline, point position and depth changes the triangle's size but preserves every viewing angle: 100×1/8=12.5 and 100×(1−2)/8=−12.5, exactly as before. The highlighted base identifies the smaller reconstruction.

Calibrated monocular SfM therefore recovers geometry up to a global similarity transform. Calling the baseline “one” chooses a reconstruction unit; it does not establish a meter. A known distance, metric sensor or other external constraint supplies scale.

Register Another Camera

Once a seed pair supplies 3D points, a new image can be registered from 3D-to-2D correspondences using a perspective-n-point pose estimator, usually robustly. The squares are known points; ticks are their projected image positions. This flattened schematic omits one spatial dimension, and is not a recommended coplanar solver configuration.

After registration, tracks seen from the new camera can support additional triangulation. Structure-from-Motion Revisited details initialization, registration and refinement; each stage can fail on weak or inconsistent evidence.

Measure Pixel Residuals

Project the estimated 3D point through its estimated camera, then compare that predicted pixel with the detected observation. In this separate magnified fixture, prediction (10,10) and observation (13,14) differ by (3,4), so the Euclidean residual is sqrt(3²+4²)=5 pixels. Image v increases downward.

The hollow circle is predicted; the filled circle is observed. The diagonal is the error vector, not a scene ray. A small pixel residual alone does not prove correct scale or correct scene identity.

Refine Cameras And Points

Bundle adjustment minimizes reprojection costs over camera parameters and 3D point coordinates together, with a gauge fixed to remove arbitrary global motion and scale. The triangle and diamond identify the two variable groups; lower stems compare residual magnitudes at one common scale.

The illustrative before norms 3 and 4 give squared cost 9+16=25; after norms 1 and 2 give 1+4=5. These are a checked illustration of the objective, not an actual solver trajectory or a guarantee of convergence. Robust losses can reduce outlier influence. Joint optimization costs more than treating poses as fixed.

Where It Breaks

Rotate a camera without moving its optical center. After accounting for orientation, an observed feature still defines the same world ray. Every amber circle is an alternative depth along that ray; there is no translation baseline to choose among them.

Pure rotation can support a panorama homography but cannot supply ordinary two-view triangulation. Move laterally to obtain parallax. Very small baselines, moving objects, textureless surfaces and wrong repeated-pattern tracks can also undermine SfM. For rotation-only imagery, image stitching may match the goal better than 3D reconstruction.

The Quick Version

• Overlapping views constrain shared points and camera poses.
• Tracks join observations of the same feature across images.
• Epipolar geometry rejects inconsistent candidate matches.
• Positive depth helps select a physically valid relative pose.
• Translated calibrated rays triangulate scene depth.
• Monocular scale requires an external metric constraint.
• Existing 3D points help register new cameras.
• Reprojection residuals compare predicted and observed pixels.
• Bundle adjustment refines cameras and points together.
• Pure rotation cannot determine depth from translation parallax.