Bundle Adjustment for Maps and Poses

Initial cameras and points are roughly right but disagree by pixels. Bundle adjustment nudges all of them together until every 3D point lands on its photos.

Rays bend as cameras and points shift together until reprojection error shrinks

Why Does This Exist?

Triangulation builds each point from fixed cameras, but the cameras themselves came from noisy matches too. Fixing one while freezing the other leaves pixels on the table. Bundle adjustment exists to solve both at once: it is the final polish inside every structure from motion and visual SLAM backend, minimizing the same reprojection error everywhere instead of locally.

This page covers the joint cost and its sparsity trick. Single-point lifting lives in triangulation; drift repair across revisits lives in loop closure and pose graph optimization.

Think of It Like This

Tightening a tent with all ropes at once

A tent sags because no single rope is right. Pulling one rope tight just moves the sag elsewhere. You walk around loosening and tightening every rope a little until the canvas sits flat and every peg agrees.

Cameras are pegs, 3D points are canvas corners, and reprojection error is the slack. The analogy stops at convexity: tents settle anywhere flat while reprojection cost has narrow valleys that need damping to enter.

How It Actually Works

The cost sums squared reprojection residuals over every observation: $C = \sum_{i,j} \|x_{ij} - \pi(P_i, X_j)\|^2$ , where $x_{ij}$ is the observed pixel of point $j$ in camera $i$ and $\pi$ projects with camera $P_i$ . Each residual touches exactly one camera and one point, so the Gauss-Newton Hessian has a block-arrow shape: dense camera-camera blocks stay small while the huge point-point block is block-diagonal and cheap to invert. The Schur complement eliminates points first, solves the small camera system, then back-substitutes for points. Levenberg-Marquardt damping handles the valley floors, and a robust loss such as Huber down-weights mismatches.

Worked numbers

Ten points start with mean residual $2.4$ px, so $C = 10 \times 2.4^2 = 57.6$ . One damped step moves each camera rotation by $0.3$ degrees and each point by $0.02$ m, cutting mean residual to $0.9$ px ( $C = 8.1$ ). A second step reaches $0.35$ px ( $C = 1.2$ ). The $47$ -fold cost drop comes from moving cameras and points together; moving points alone stalls near $1.1$ px because the camera bias remains.

Code

1def reprojection_cost(residuals: list[float]) -> float:2    return sum(r * r for r in residuals)  # sum of squared pixel errors3
4
5print(reprojection_cost([2.4] * 10))  # -> 57.66print(reprojection_cost([0.35] * 10))  # -> 1.225

Watch Out For

Running full bundle adjustment on every frame

Global solving over thousands of frames grows with the cube of the camera count without the Schur trick, and even with it, per-frame global solves miss real time by orders of magnitude. The symptom is a tracker that starts smooth then lags seconds behind. Use local windows per frame and save the global solve for loop closures.

Leaving outliers in the squared cost

One $20$ px mismatch contributes $400$ to $C$ and drags good cameras toward it. The map bends to please the worst match. Wrap residuals in a Huber loss and re-check after solving, or the adjustment optimizes the mistakes.

The Quick Version

Bundle adjustment refines all cameras and all points against one reprojection cost.
The cost is the sum of squared pixel residuals across every observation.
Sparsity plus the Schur complement makes the large system solvable.
Damping and a robust loss keep the solver inside narrow valleys.
Mean error typically falls from pixels to sub-pixel in a few steps.
Solve locally per frame and globally only at loop closures.