Visual SLAM

A camera dropped in an unknown building must draw the map and track itself inside it at the same time. Visual SLAM does both from images alone, then snaps the map shut when it recognizes a place it has seen.

Why Does This Exist?

GPS fails indoors, wheel encoders slip, and prebuilt maps go stale the moment furniture moves. A robot or headset with only a camera faces a chicken-and-egg task: it needs a map to know where it is, and it needs to know where it is to build the map. Visual SLAM solves both at once by tracking how pixels move across frames, triangulating the world behind them, and recognizing revisited places to cancel accumulated drift.

This page covers the full loop: tracking, mapping, and closure. The frame-to-frame front end is visual odometry; the offline cousin is structure from motion; the joint refinement is bundle adjustment.

Think of It Like This

Exploring a dark building with a flashlight

You wake in an unfamiliar building with only a flashlight. You note a chair, step forward, and watch the chair slide in the beam: the shift tells you both where the chair stands and how far you stepped. Repeating this for hundreds of landmarks builds a floor plan while keeping you located inside it. Returning to the lobby and recognizing it lets you fix every mismeasured corridor at once.

Landmarks are tracked visual features, the floor plan is the point map, and the lobby moment is loop closure. The analogy stops at sensing: you feel your steps through balance, while a pure visual system has no step sense and must infer every motion from pixels.

How It Actually Works

Tracking, mapping, loop closure

Tracking matches features from the last keyframe into each new frame and solves for the pose that minimizes reprojection error. Mapping triangulates new 3D landmarks $X_j$ from matched 2D observations $x_{ij}$ as the camera moves. Bundle adjustment refines poses and landmarks jointly by minimizing $\sum_{i,j} \|x_{ij} - \pi(K(R_i X_j + t_i))\|^2$ , where $R_i$ and $t_i$ are camera $i$ 's rotation and translation, $K$ holds the intrinsics, and $\pi$ is perspective projection. Loop closure watches for revisited places with appearance matching, verifies them geometrically, and runs a pose-graph optimization that spreads the accumulated gap across the whole trajectory instead of leaving it at the seam.

Worked loop numbers

A robot drives a 400 m loop with 1 percent odometry drift, so the returning pose sits $400 \times 0.01 = 4.0$ m from the mapped start. Recognizing the start closes the loop, and the pose graph spreads that 4 m gap over 400 keyframes at $4.0/400 = 0.01$ m, about 1 cm of correction each. Refinement quality shows in pixels too: three residuals of $[1.2, 0.8, 2.0]$ px carry an RMS of $\sqrt{(1.44 + 0.64 + 4.00)/3} = \sqrt{2.03} = 1.42$ px, and bundle adjustment shrinking them to $[0.3, 0.2, 0.4]$ px leaves $\sqrt{(0.09 + 0.04 + 0.16)/3} = \sqrt{0.097} = 0.31$ px.

Code

1import math2
3before = [1.2, 0.8, 2.0]4after = [0.3, 0.2, 0.4]5print(round(math.sqrt(sum(r * r for r in before) / 3), 2))6# -> 1.427print(round(math.sqrt(sum(r * r for r in after) / 3), 2))8# -> 0.319print(round(4.0 / 400, 3))10# -> 0.01

Watch Out For

Featureless or dynamic scenes

Blank walls, mirrors, and crowds of moving people give tracking nothing stable to hold. The symptom is a sudden track loss followed by relocalization jumps. Point the system at textured static structure, mask known moving regions, and treat every dynamic landmark as guilty until proven static.

False loops from repeated structure

Identical corridors and parking floors trigger appearance matches between places that only look alike. The symptom is a map that snaps shut at the wrong seam and bends the trajectory. Demand geometric verification with enough inlier matches before any closure updates the graph.

The Quick Version

Visual SLAM builds the map and tracks the camera inside it at the same time, from images alone, providing base pose estimates for robot localization and mapping.
Tracking solves each pose from reprojection error; mapping triangulates the landmarks behind it.
Bundle adjustment minimizes squared reprojection error over poses and landmarks jointly.
A 4 m gap over a 400 m loop spreads to about 1 cm of correction per keyframe at closure.
Textureless and dynamic scenes starve tracking; repeated structures demand verified closures.