Neural Radiance Fields

Instead of explicitly storing 3D shapes, NeRF trains a neural network to memorize a 3D scene and render new views from any angle.

Querying a neural network for color and density along a camera ray

Why Does This Exist?

In traditional 3D graphics, scenes are represented explicitly using meshes (polygons), point clouds, or voxels. If you want to render an image of an object from a new angle (novel view synthesis), you build an explicit 3D model from your 2D photos first.

However, extracting highly detailed meshes from photos is famously difficult, especially for complex lighting, reflections, or semi-transparent materials. Neural Radiance Fields (NeRFs) introduced a radical alternative: what if we don't build a 3D shape at all, and instead train a neural network to perfectly answer the question, "What color and density exists at this exact $(x, y, z)$ point in space when viewed from this angle?"

Think of It Like This

The Hyper-Specific Librarian

Imagine a massive library representing a 3D room.

The traditional mesh approach is like building a miniature architectural model of the room out of cardboard and painting it. If someone asks what it looks like from the doorway, you hold up your cardboard model and look at it from that angle.

The NeRF approach is like hiring an eccentric librarian who has memorized the entire room. You don't have a physical model. Instead, you shine a laser pointer into the empty space and constantly ask the librarian: "At this exact inch along the laser beam, is there something solid? And if so, what color is it?" By summing up the librarian's answers along millions of laser beams, you can paint a perfect picture of the room from any doorway.

How It Actually Works

A NeRF represents a 3D scene as a continuous mathematical function, approximated by a Multilayer Perceptron (MLP).

1. Shooting Rays

To render an image from a specific camera position, we shoot mathematical "rays" from the camera through each pixel of the virtual image plane and into the 3D scene.

2. Querying the MLP

Along each ray, we sample multiple 3D points $(x, y, z)$ . For every sampled point, we feed its coordinates and the ray's viewing direction $(\theta, \phi)$ into the MLP. The MLP outputs two things:

Color $(r, g, b)$ : The emitted color at that point.
Volume Density $(\sigma)$ : How opaque or solid that point is.

3. Volume Rendering

Because the MLP gives us color and density at discrete points along the ray, we can use classical volume rendering techniques to combine them. We mathematically "march" along the ray, accumulating the color of each point based on how dense it is and how much light was blocked by points in front of it. This produces the final pixel color.

4. Training

The model is trained entirely by comparing these rendered pixels to real 2D photos taken from known angles. If the rendered pixel is red but the actual photo shows blue, the loss function updates the MLP's weights to adjust the color and density of the points along that ray.

Code

Here is a conceptual snippet showing how a simplified NeRF MLP takes inputs and returns color and density.

1import torch2import torch.nn as nn3
4class SimpleNeRF(nn.Module):5    def __init__(self):6        super().__init__()7        # Position network determines density and intermediate features8        self.pos_net = nn.Sequential(9            nn.Linear(3, 256), nn.ReLU(),10            nn.Linear(256, 256), nn.ReLU(),11            nn.Linear(256, 256 + 1) # Outputs 256 features + 1 density12        )13        14        # Color network takes viewing direction and position features15        self.color_net = nn.Sequential(16            nn.Linear(256 + 3, 128), nn.ReLU(),17            nn.Linear(128, 3), nn.Sigmoid() # Outputs RGB18        )19        20    def forward(self, position, direction):21        # position: (x, y, z)22        # direction: (theta, phi) or normalized vector23        24        # 1. Get density and features purely from spatial position25        pos_out = self.pos_net(position)26        features, density = pos_out[..., :-1], pos_out[..., -1:]27        density = torch.relu(density) # Density cannot be negative28        29        # 2. Get color by combining features and viewing direction30        # This allows view-dependent effects like reflections!31        color_input = torch.cat([features, direction], dim=-1)32        rgb = self.color_net(color_input)33        34        return rgb, density

Watch Out For

Extreme Compute Costs

The original NeRF is notoriously slow. Rendering a single 1080p image requires shooting millions of rays and sampling dozens of points per ray. This means evaluating the MLP tens of millions of times per frame. While newer techniques (like 3D Gaussian Splatting) have made real-time rendering possible, basic NeRFs are incredibly computationally expensive.

The Quick Version

NeRF stands for Neural Radiance Field.
It does not use explicit 3D geometry (like meshes); the 3D scene is memorized within the weights of a neural network.
The MLP takes a 3D coordinate and viewing direction as input, and outputs color and density.
Images are rendered by shooting rays through pixels and accumulating the MLP's outputs along the ray.
It handles reflections and complex lighting beautifully because color depends on the viewing angle.