Marigold Diffusion Depth Model

Start from image noise and denoise it into depth. Marigold reuses a text-to-image diffuser as a sharp monocular depth painter.

Noise denoises step by step into sharp depth under image conditioning

Why Does This Exist?

Regressive depth nets blur edges because mean losses prefer safe middles. Marigold exists to keep the sharp prior of large image diffusers: by finetuning latent diffusion on depth pairs, it samples crisp occlusion boundaries and fine rails that discriminative losses wash out. It trades speed for edge quality on single stills.

Foundations are monocular depth estimation; the discriminative scale alternative is Depth Anything.

Think of It Like This

Developing a photo in trays

A print starts gray in developer, then edges rise first and tones settle with time. Pull it early for soft shapes; leave it full term for crisp lines.

Noise is the gray sheet and denoising steps are the trays. The analogy stops at randomness: trays develop deterministically while diffusion sampling jitters each run, which is why ensembling helps.

How It Actually Works

The image encodes to latents that condition a UNet denoiser trained with $z_t = \sqrt{\bar{\alpha}_t} z_0 + \sqrt{1 - \bar{\alpha}_t} \epsilon$ on depth latents $z_0$ from synthetic sets. Inference runs about $10$ DDIM steps from pure noise per sample, then decodes and median-aligns to affine depth. Test-time ensembling over $10$ seeds takes the pixelwise median, which trims stochastic spikes at edges while keeping the sharp mode that regression averages away.

Worked numbers

One sample gives edge mean error $0.42$ m on the demo split; median of $10$ samples gives $0.34$ m, about a $20$ percent cut. Full inference costs $10$ steps times $10$ samples $= 100$ UNet passes versus one pass for DPT, so Marigold suits offline stills rather than video rate.

Code

1def ensemble_median(runs: list[float]) -> float:2    ordered = sorted(runs)  # median trims diffusion outliers3    return ordered[len(ordered) // 2]4
5
6print(ensemble_median([0.41, 0.34, 0.38, 0.33, 0.36]))  # -> 0.36

Watch Out For

Using one Marigold sample as truth

Single diffusion draws jitter by several percent on textureless fills. The symptom is flickering depth across reruns of the same photo. Always ensemble near ten samples and report the median, or pick a deterministic regressor for video.

The Quick Version

Marigold finetunes latent diffusion from image generation into depth.
Conditioning on the photo steers denoising toward consistent geometry.
Samples keep sharper edges than mean-regression outputs.
Median over about ten runs trims stochastic edge spikes.
Cost is roughly one hundred UNet passes per final map.
Use it for crisp stills, not real-time streams.