DreamFusion Text to 3D Objects

DreamFusion lifts a 2D image diffusion model into 3D by using its denoising score to sculpt a NeRF from text alone.

Why Does This Exist?

Paired text plus 3D data is scarce while text plus images is vast. Training text-to-3D from scratch starves. DreamFusion (2022) freezes a 2D diffusion model and uses score distillation sampling (SDS): render the current NeRF from a random view, ask diffusion how to denoise it toward the prompt, and backpropagate that direction into 3D. No 3D supervision needed.

This page covers SDS lifting. Prompt-level guidance lives in text to 3D.

Think of It Like This

A sculptor checked by a photo critic

A sculptor cannot see the final statue clearly. After each chisel stroke they photograph it from a random angle and show a critic who only knows photos. The critic says how to make this photo match the description better, and the sculptor translates that into the next 3D cut.

NeRF is the clay and diffusion is the critic. The analogy stops at views: the critic sees many random angles, not one favorite.

How It Actually Works

NeRF parameters $\theta$ render image $x = g(\theta, v)$ at random view $v$ . Diffusion adds noise $\epsilon$ at level $t$ and predicts $\hat{\epsilon}(x_t, y)$ for prompt $y$ . SDS gradient is $\nabla_\theta \mathcal{L} = \mathbb{E}[w(t)(\hat{\epsilon} - \epsilon) \partial x / \partial\theta]$ . Steps use high guidance weights around $100$ and anneal $t$ from noisy to clean.

Worked example

Prompt is a squirrel astronaut. Step 1,000 renders a gray blob at $v = 30$ degrees azimuth. Diffusion at $t = 0.6$ predicts noise pointing toward visor highlights with residual norm $2.3$ . The SDS step moves density $+0.4$ at the visor shell and $-0.2$ at the back. After $10{,}000$ steps the shell visor reads $[0.9, 0.9, 0.95]$ while the suit stays $[0.55, 0.40, 0.30]$ .

Watch Out For

Janus faces from view-independent prompts

Prompting a squirrel astronaut without view tags grows faces on all sides. The symptom is three visors around one helmet. Append view tags like front view and side view per sampled camera.

The Quick Version

DreamFusion reuses frozen 2D diffusion to supervise 3D without 3D data.
Random views plus SDS gradients sculpt the NeRF toward the prompt.
High guidance near $100$ keeps shapes prompt-faithful.
Do not expect exact measurements; prompts underspecify scale and parts.
View-tagged prompts suppress duplicated faces around the shape.