NeRF in the Wild for Photo Sets

NeRF in the Wild learns per-photo lighting and ignores tourists so landmark photos still build one clean 3D scene.

Why Does This Exist?

Lab NeRFs assume one lighting and no people. Tourist photo sets break both: noon versus dusk changes every color, and pedestrians block facades. Filtering photos by hand does not scale. NeRF in the Wild (2020) adds a per-image appearance code plus a transient density branch, so one static scene explains all photos while lighting and tourists factor out.

Think of It Like This

One statue sketched in changing weather with crowds

An artist sketches one statue across sunny, cloudy, and night visits while tourists wander through. The statue lines stay identical across sketches; only the shading notes and the crossed-out tourist blobs change per visit.

Static weights are the statue lines, appearance codes are shading notes, transient heads are crossed-out blobs. The analogy stops at optimization: all three learn jointly, not in separate passes.

How It Actually Works

Static branch outputs $(\sigma, c)$ from position and direction plus appearance code $\ell_i$ of dim $48$ per photo $i$ . Transient branch outputs $(\sigma_i^T, c_i^T)$ from position plus a transient code. Final color blends static and transient by their densities along the ray. Loss adds uncertainty weighting so tourist pixels down-weight automatically.

Worked example

Photo A at noon has $\ell_A = [0.3, 0.1]$ and photo B at dusk has $\ell_B = [-0.4, 0.2]$ . A facade point renders static albedo $[0.70, 0.65, 0.60]$ , shifted to $[0.78, 0.72, 0.66]$ under $\ell_A$ and $[0.45, 0.42, 0.50]$ under $\ell_B$ . A tourist pixel gets transient density $3.0$ versus static $0.2$ , so $94$ percent of its color comes from the transient head and the static wall stays clean.

Watch Out For

Appearance codes memorizing geometry

A $128$ -dim code with few photos per code bakes wall cracks into lighting. The symptom is novel views with ghost shadows. Keep codes near $48$ dims and share them across hundreds of photos.

The Quick Version

Per-photo codes absorb lighting so one static scene fits all photos.
Transient heads absorb people and cars per ray.
Uncertainty weighting mutes tourist pixels automatically.
Do not use it on single-lighting lab captures; plain NeRF trains faster.
Static albedo plus small appearance shifts explain day-to-night sets.