Structure from Motion

Reconstructing a 3D scene and the camera's path simultaneously from a moving video or a collection of images.

Why Does This Exist?

If you have a single photo of a building, it's flat. But if you walk around the building taking pictures from different angles, your brain easily understands the 3D shape. Structure from Motion (SfM) exists to give computers this same ability. Instead of requiring specialized hardware like LiDAR or stereo depth cameras, SfM can build dense 3D models and figure out exactly where the camera was located just by looking at how features shift across a standard 2D image sequence. It's the foundation of modern photogrammetry, Google Earth 3D views, and drone mapping.

Think of It Like This

Closing your eyes on a train

Imagine sitting on a moving train. You look out the window at a distant mountain and a nearby fence. As you move, the fence zips by incredibly fast, while the mountain barely seems to move at all (motion parallax).

If you recorded the exact speed each object appeared to move across your vision, you could work backwards to figure out two things simultaneously:

Exactly how far away the fence and mountain are (the structure).
Exactly how fast your train is moving (the motion).

SfM does this mathematically across thousands of visual landmarks.

How It Actually Works

The SfM pipeline generally follows these steps:

Feature Extraction and Matching: Algorithms like SIFT or SuperPoint find distinct features (corners, blobs) in every image. The system then matches these features across different images to track the same physical point from multiple viewpoints.
Camera Pose Estimation: Using a subset of images (often starting with a stereo pair), it estimates the relative camera positions and orientations (the "motion") using Epipolar Geometry.
Triangulation: Once camera positions are known, the 2D positions of a matched feature in multiple images can be projected out as rays. Where these rays intersect in 3D space defines the location of that physical point (the "structure").
Bundle Adjustment: Because small errors in tracking multiply rapidly, SfM uses a massive optimization step called Bundle Adjustment. It simultaneously refines all 3D point coordinates and all camera parameters to minimize the reprojection error—the difference between where a 3D point actually appears in the 2D image and where the math says it should appear.

Code

1# Pseudo-code for a standard SfM pipeline2images = load_images()3features = [extract_features(img) for img in images]4matches = match_features(features)5
6# Initialize with the best pair7camera_poses, point_cloud = initialize_sfm(images[0], images[1], matches)8
9for img in images[2:]:10    # Find where this new camera is based on known 3D points11    new_pose = solve_pnp(img, point_cloud, matches)12    camera_poses.append(new_pose)13    14    # Triangulate new 3D points15    new_points = triangulate(camera_poses, matches)16    point_cloud.add(new_points)17    18    # Globally optimize everything19    camera_poses, point_cloud = bundle_adjustment(camera_poses, point_cloud)

Watch Out For

Pure Rotation

If the camera only rotates (like on a tripod) and doesn't translate (move through space), you get no motion parallax. SfM completely fails to estimate depth in this scenario because there is no baseline between shots.

Scale Ambiguity

SfM can reconstruct the shape of the scene perfectly, but it cannot know the absolute size. A toy car shot up close looks mathematically identical to a real car shot from far away. You need external markers (like a GPS or an object of known size) to establish real-world scale.

The Quick Version

Structure from Motion builds 3D models from moving 2D cameras.
It simultaneously calculates the structure (3D point cloud of the scene) and the motion (the trajectory of the camera).
It relies on feature matching across images and triangulating rays.
Bundle Adjustment is the secret sauce that globally optimizes the math to keep the model from drifting apart.