Structure from Motion

Overlapping photos of one scene reveal both the 3D point positions and the camera path, because each matched feature triangulates from several viewpoints at once.

Three camera positions sight the same point from different angles, and intersecting those rays fixes both the 3D point and where each camera stood.

Why Does This Exist?

One photo is flat, but a walk around a building with a phone produces a 3D model on commodity hardware with no laser scanner. Structure from Motion exists to give computers that walk-around understanding: from overlapping ordinary photos it recovers the scene points, called the structure, and the camera positions, called the motion, at the same time. It powers photogrammetry, drone mapping, and 3D views assembled from crowd photos.

The optimization core has its own page in bundle adjustment, the correspondence front end in feature matching, and the ray-intersection math in triangulation. This page covers the pipeline order, the projection equation with numbers that expose scale ambiguity, and the two motion conditions that silently break reconstructions.

Think of It Like This

Closing your eyes on a train

Ride a train past a fence with mountains behind. The fence zips by while the mountains crawl, and the speed difference across your window encodes both how far each object sits and how fast you travel. Record every apparent speed precisely and you can solve for the distances and the train's motion together.

Matched features are the fence posts and peaks, apparent speed is the pixel shift between photos, and the joint solve is the pipeline below.

The analogy stops at the window. A train moves smoothly along rails, while photo collections jump between unknown positions with unknown gaps, so the algorithm must discover the motion itself instead of reading it off a speedometer.

How It Actually Works

The pipeline order

Features such as corners are extracted per image and matched across views, so one physical point owns a track of 2D observations. Starting from an initial image pair, relative camera poses come from epipolar geometry, and each matched track triangulates into a 3D point where its rays intersect. New images register against the points already built, adding cameras and points incrementally. Finally bundle adjustment refines everything at once: all 3D points plus all camera parameters shift together to minimize reprojection error, the pixel gap between each observed feature and where the current model says it should project.

The projection equation and scale ambiguity

A pinhole camera maps a point at lateral position $X$ and depth $Z$ to pixel $x = f \cdot X / Z$ , where $f$ is focal length in pixels. Take $f = 500$ : a toy marker with $X = 0.1$ meters at $Z = 1$ meter projects to $x = 500 \cdot 0.1 / 1 = 50.0$ pixels. A real marker with $X = 1$ meter at $Z = 10$ meters projects to $x = 500 \cdot 1 / 10 = 50.0$ pixels. Identical observations, tenfold size difference: images alone fix shape but never absolute scale, so real-world size needs GPS, an inertial unit, or a known-size reference.

Worked example: pricing one reprojection error

Suppose the current model places a point and camera so the point should project to pixel $102.0$ , but the matched feature sits at $99.7$ . The reprojection error for that observation is $102.0 - 99.7 = 2.3$ pixels, contributing $2.3^2 = 5.29$ to the squared-error sum. Bundle adjustment sums terms like this over thousands of observations and descends until typical residuals fall below one pixel. One $2.3$ -pixel term is routine noise; hundreds of them all leaning one direction signal a misregistered camera rather than bad luck.

Code

1def project(x_meters, z_meters, focal_px=500.0):2    """Pinhole projection of a lateral position to pixel coordinate."""3    return focal_px * x_meters / z_meters4
5
6toy = project(0.1, 1.0)7real = project(1.0, 10.0)8print(toy, real)9# -> 50.0 50.010
11error = 102.0 - 99.712print(round(error, 1), round(error ** 2, 2))13# -> 2.3 5.29

Watch Out For

Shooting with rotation but no translation

A camera spinning on a tripod gives views with no baseline, hence no parallax for triangulation to grip. The symptom is a reconstruction that initializes and then collapses or refuses to start. Move the camera through space between shots so every point is sighted from genuinely separated positions.

Reading meters off a scaleless model

The pipeline reconstructs shape perfectly while scale floats freely: the toy-versus-real example above proves identical pixels can mean sizes tenfold apart. The symptom is a beautiful model with wrong dimensions, discovered only at measurement time. Pin scale with GPS, an inertial track, or one known distance before quoting any size.

The Quick Version

Matched features across overlapping photos triangulate into 3D points and camera poses jointly.
Projection follows $x = f \cdot X / Z$ , and identical pixels can hide tenfold scale differences.
Bundle adjustment minimizes summed squared reprojection error across all observations.
A $2.3$ -pixel residual contributes $5.29$ to that sum, noise alone but a signal in bulk.
Pure rotation without translation gives no parallax, and scale needs an external reference.