Novel View Synthesis from Photos

Novel view synthesis renders unseen viewpoints by interpolating learned scene structure instead of copying pixels.

Why Does This Exist?

Shoppers, drivers, and robots all want the angle that was never captured. Copying the nearest photo ghosts at occlusions, while rebuilding full CAD per scene costs too much. NeRF and multi-plane images both serve this task with different speed and baseline trade-offs.

Think of It Like This

Between two windows onto a street

Two windows show a street from slightly different spots. Standing between them you can imagine the middle view well. Stepping far past either window forces guessing what hides behind the corner shop.

Captured photos are windows and new views are standing spots. The analogy stops at learning: models blend features, not mental images.

How It Actually Works

Given views $\{I_i, P_i\}$ with poses $P_i$ , synthesis renders $I_{new}$ at $P_{new}$ by warping, blending, or volumetric marching. Quality follows baseline ratio $b = \lVert P_{new} - P_{near} \rVert / \text{scene depth}$ . Interpolation with $b < 0.1$ stays clean; extrapolation past $b > 0.5$ tears.

Worked example

Ten photos ring a chair at $36$ -degree steps. A new view $10$ degrees from the nearest capture has $b = 0.08$ at $2$ m depth and renders with PSNR near $28$ dB. A view $90$ degrees past the last capture has $b = 0.7$ and drops near $19$ dB with smeared legs. The $9$ dB gap is the honest price of extrapolation.

Watch Out For

Judging synthesis by training views only

Reporting PSNR on held-out crops of training poses hides extrapolation failure. The symptom is demos that collapse on first user drag. Always test midway between captures and past the ring edge.

The Quick Version

Synthesis renders unseen angles from captured photos plus poses.
Interpolation between captures stays photoreal with dense rings.
Extrapolation past the capture ring tears and smears quickly.
Do not promise orbit freedom from two photos; add captures or priors.
Baseline ratio predicts quality better than model size alone.