Skip to content
AI360Xpert
Paper Breakdowns
Paper breakdown

3D Gaussian Splatting

The 2023 paper that dethroned NeRFs for novel view synthesis by replacing the neural network with millions of explicit 3D splats, achieving real-time 1080p rendering.

Paper: 3D Gaussian Splatting for Real-Time Radiance Field Rendering

Authors: Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, George Drettakis · 2023

Read the paper
Unlike NeRFs which query a neural network for every pixel, Gaussian Splatting explicitly sorts and projects 3D ellipses (Gaussians) onto the 2D screen, running at 100+ FPS.
Unlike NeRFs which query a neural network for every pixel, Gaussian Splatting explicitly sorts and projects 3D ellipses (Gaussians) onto the 2D screen, running at 100+ FPS.

The Problem

Neural Radiance Fields (NeRFs) were beautiful but hopelessly slow. To render a single HD image, a NeRF had to evaluate a neural network millions of times (ray marching). Even highly optimized versions took seconds per frame. The graphics industry needed a method that offered the photorealism of NeRFs but operated using explicit geometry that modern GPUs could actually rasterize in real-time (60+ FPS).

The Idea

Researchers at Inria replaced the neural network entirely. Instead of an implicit function, they represent a scene using an explicit cloud of millions of 3D Gaussians (blobs). Each blob has a position, covariance (shape/size), opacity, and color (represented by Spherical Harmonics to allow colors to change based on viewing angle). Because the math of projecting a 3D Gaussian onto a 2D screen is analytically simple, they built a highly optimized GPU rasterizer to draw them instantly.

How It Works

The training process is a complex choreography of geometry:

  1. Initialization: Start with a sparse point cloud generated from the 2D photos (using standard Structure from Motion). Turn each point into a 3D Gaussian.
  2. Rendering (Forward Pass): For a given camera angle, sort all the Gaussians from front-to-back. Mathematically project them onto the 2D image plane (a process called "splatting"). Blend them using alpha compositing.
  3. Optimization: Calculate the loss against the ground truth photograph and backpropagate to update the position, shape, color, and opacity of every Gaussian.
  4. Adaptive Density Control: During training, if a Gaussian is too large, split it into two smaller ones. If a Gaussian is useless (transparent), delete it. This allows the scene to "grow" detail exactly where needed.

Why It Mattered

3D Gaussian Splatting (3DGS) fundamentally changed the trajectory of view synthesis. By abandoning the neural network and returning to explicit rasterization, it achieved real-time 1080p rendering (100+ FPS) while maintaining or exceeding NeRF quality. It bridged the gap between AI research and the traditional video game graphics pipeline.

What Came After

3DGS rapidly replaced NeRFs in almost all commercial applications (like Luma AI and Polycam). Subsequent research focused on animating the Gaussians (4D Gaussian Splatting for video) and compressing the massive file sizes (which can exceed gigabytes for a single scene).