Self-Supervised Depth from Video

No lidar, no labels: warp a nearby frame into the current one and learn depth from the mismatch. Video itself becomes the teacher.

Neighbor frames warp by depth and pose to teach the current frame

Why Does This Exist?

Labeled depth needs expensive lidar rigs, yet unlabeled driving video is endless. Self-supervised depth exists to train from that video alone: a depth net plus a pose net synthesize one frame from its neighbors, and brightness mismatch becomes the loss. Monodepth2 made the recipe robust with masking and multiscale sampling.

Teacher-free basics build on monocular depth estimation; motion comes from visual odometry.

Think of It Like This

Learning sculpture by shadow matching

Shine a lamp on clay from two angles and carve until both shadows match the photos. No ruler guides the knife; shadow agreement alone shapes the clay.

Depth is the clay shape and warping is the shadow test. The analogy stops at moving objects: lamps never move while cars do, breaking the static-world deal.

How It Actually Works

Given target $I_t$ and source $I_s$ , predicted depth $\hat{D}_t$ and relative pose $\hat{T}$ warp source into target coordinates, then bilinear-sample $\hat{I}_t$ . Loss blends $L1$ with structural similarity: $L = 0.85 \times (1 - \mathrm{SSIM})/2 + 0.15 \times |I - \hat{I}|$ , minimized over the best of two neighbors per pixel. Automasking drops pixels where warped error exceeds unwarped error, removing static frames and same-speed cars. Median scaling at test time fixes the inherent scale ambiguity.

Worked numbers

A patch with $L1$ error $0.10$ and SSIM $0.90$ gives $L = 0.85 \times 0.05 + 0.15 \times 0.10 = 0.0425 + 0.015 = 0.0575$ . Good driving frames average near $0.08$ ; dynamic-car pixels score $0.25$ before automasking removes them, cutting their share from $12$ percent to $2$ percent of the loss.

Code

1def photo_loss(l1: float, ssim: float) -> float:2    return 0.85 * (1.0 - ssim) / 2.0 + 0.15 * l1  # Monodepth2 blend3
4
5print(round(photo_loss(0.10, 0.90), 4))  # -> 0.0575

Watch Out For

Training through moving objects

Cars at matched speed look static, so the net paints them at infinite depth to explain zero parallax. The symptom is holes of far depth on every vehicle. Automask plus motion-aware losses are required on street video, not optional extras.

The Quick Version

Self-supervised depth learns from view synthesis, not lidar labels.
Depth plus pose warp neighbors into the target for a photo loss.
SSIM plus $L1$ with per-pixel minimum over neighbors sets the standard.
Automasking removes static and co-moving pixels automatically.
Scale stays ambiguous and needs median alignment at test time.
Moving objects are the main failure without masking.