NeRF
The 2020 paper that revolutionized 3D computer vision by proving a neural network could "memorize" a 3D scene and render photorealistic novel views.
Paper: NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
Authors: Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng · 2020
Read the paper
The Problem
Creating photorealistic 3D models from a set of 2D photographs (photogrammetry) was historically done using discrete geometry—creating a massive 3D mesh of polygons and projecting textures onto them. This process was complex, computationally heavy, and struggled immensely with reflections, transparency (like glass or smoke), and fine details (like hair).
The Idea
UC Berkeley and Google researchers proposed Neural Radiance Fields (NeRF). Instead of building a discrete 3D mesh, they trained a Multi-Layer Perceptron (MLP) neural network to act as a continuous, implicit 3D scene. The network acts as a function: you input a 3D coordinate (x, y, z) and a viewing angle, and the network outputs the color (RGB) and the density (opacity) at that exact point in space.
How It Works
To render a 2D image from a new viewpoint using a NeRF:
- Ray Marching: For every pixel in the desired 2D image, shoot a "ray" straight out into the 3D scene.
- Sampling: Sample hundreds of specific 3D points along that ray.
- Network Query: Feed the (x,y,z) coordinates of those points and the viewing direction into the neural network to get the color and density for each point.
- Volume Rendering: Use classical volume rendering equations to accumulate those colors and densities along the ray to determine the final color of the 2D pixel.
To train the network, you use gradient descent to minimize the error between the rendered pixels and the actual pixels of the training photographs.
Why It Mattered
NeRF was a massive breakthrough because it seamlessly handled view-dependent effects (like a shiny reflection that moves as you walk around it) and semi-transparent objects. It spawned an entire sub-field of computer vision overnight, transitioning 3D modeling from discrete geometry into continuous neural functions.
What Came After
While the original NeRF produced stunning results, it was glacially slow (taking hours to train and minutes to render a single frame, since it required millions of network queries per image). The next three years of research focused entirely on speeding it up (e.g., Instant NGP), eventually culminating in 3D Gaussian Splatting, which largely replaced NeRFs for real-time rendering.