Light Field Imaging

A photo records how much light arrived, forgetting where it came from. Light field cameras record direction too, so focus and viewpoint become software choices after the shot.

Ten angular samples per pixel refocus after capture, trading a 16 megapixel sensor for 0.16 megapixel views

Why Does This Exist?

Conventional capture bakes decisions in glass: focus plane, aperture, viewpoint — set at exposure, regretted forever. Missed focus ruins portraits; fixed stereo baselines limit depth range; a single viewpoint hides occluded detail. The photons carried the missing answers and the sensor discarded them.

Light field cameras keep direction. A microlens array sorts incoming rays by angle onto sensor regions, recording the 4-D field (two spatial, two angular coordinates) in one exposure. Refocus, aperture, and small viewpoint shifts become post-capture computation. The software-camera sibling is computational photography; the depth sibling is stereo vision.

The one idea you need first. Shift-and-add refocus synthesizes a focal plane by shifting angular views according to their disparity at that depth and averaging: rays from the chosen plane align and sum coherently, rays from other depths spread and blur. No deconvolution, no priors — refocus is literally re-sorting recorded rays.

Think of It Like This

Every instrument on its own track

A mono concert recording freezes the mix: loud guitar forever. A multitrack session records each instrument separately, and the mix becomes a decision made after — rebalance, mute the cough, spotlight the solo. Nothing is lost at capture except disk space.

Light fields multitrack light. Each angular sample is an instrument track; refocus is the remix, viewpoint shift is panning the balance. The price is tracks per song: angular samples consume pixels that once bought resolution, so the remix trades sharpness for freedom.

How It Actually Works

Microlenses tile the sensor; each covers an $N \times N$ pixel block recording one spatial point from $N^2$ angles. Calibration maps pixels to rays (centers, directions, vignetting). Rendering picks a virtual focus: shift each angular view by its depth-dependent disparity, sum, normalize. Viewpoint synthesis extrapolates nearby views within the baseline the array spans (millimeters — parallax for refocus and small look-arounds, not walk-throughs).

The resolution tax is fixed: spatial pixels equal sensor pixels divided by angular samples.

Worked example

Sensor $4{,}000 \times 4{,}000 = 16$ megapixels; angular sampling $10 \times 10 = 100$ directions per microlens. Spatial resolution: $400 \times 400 = 160{,}000$ pixels $\approx 0.16$ MP — a hundredfold fewer than the sensor's $16$ MP.

Refocus gain at the chosen plane with $100$ views: in-focus rays sum coherently (signal $\times 100$ , noise $\times \sqrt{100} = 10$ ), so SNR improves $10\times$ over one view. Out-of-plane rays land spread across $10$ pixel offsets and average toward haze. Focus becomes a hundred-view consensus instead of one glass verdict.

Code

The point of this snippet is the tax: sensor pixels divided by angular samples sets the refocusable resolution.

1def light_field_spatial(sensor_px: int, angular: int) -> int:2    return (sensor_px // angular) ** 2  # pixels left for the image3
4
5print(light_field_spatial(4000, 10))  # -> 1600006print(160000 / 16000000)  # -> 0.01

Watch Out For

The resolution tax arrives in every shot

$0.16$ MP views look soft next to phone photos, and no algorithm restores angular samples into spatial detail beyond modest super-resolution factors. Teams expecting phone sharpness plus refocus get neither — physics collects the tax before software speaks.

Budget the tax up front: size angular sampling to the refocus range actually needed (portrait re-focus needs fewer views than microscopy). Oversampling angles starves the image for freedom nobody uses.

Millimeter baselines, centimeter dreams

Microlens arrays span tiny baselines, so synthesized viewpoints move millimeters — enough for refocus and wiggle-3D, useless for see-behind-the-vase. Demos implying free viewpoint travel oversell the geometry.

Match baseline to task: refocus and synthetic aperture thrive; occlusion removal needs camera arrays or motion, not microlenses. Read the disparity range, not the marketing.

The Quick Version

Light field cameras record ray position plus direction, making focus and aperture post-capture choices.
Refocus is shift-and-add: chosen-plane rays sum coherently while others haze out.
The worked fixture trades $16$ MP of sensor for $0.16$ MP refocusable views at $10 \times 10$ angles.
In-focus SNR gains $10\times$ from hundred-view consensus; out-of-plane content blurs away.
Millimeter baselines serve refocus, not walk-throughs — match the rig to the task.