DETR3D Multi-View 3D Detection

DETR3D detects 3D boxes by refining a small set of 3D queries that each sample the camera views where they project.

Why Does This Exist?

Multi-camera detection duplicated boxes across overlapping views and needed non-maximum suppression to clean up. DETR removed NMS in 2D with set prediction, and DETR3D (2022) ports that to 3D: keep $Q = 900$ object queries in 3D space, project each into the views that see it, sample features there, and refine. No anchors, no NMS, no BEV grid.

This page covers query refinement. If you need a dense map for lanes and occupancy too, prefer a BEV grid method like BEVFormer.

Think of It Like This

Nine hundred scouts with radios

Nine hundred scouts stand on a field, each guessing where one object might be. Each scout radios only the camera operators who can actually see their guessed spot, listens back, and steps toward better evidence. After six rounds the scouts standing on real objects shout loudly and the rest stay quiet.

Queries are scouts and sampling is the radio. The analogy stops at matching: a Hungarian match assigns each truth to exactly one scout during training.

How It Actually Works

Query $q$ holds a 3D reference point $c = (x, y, z)$ . Each decoder layer projects $c$ into every camera with matrix $P_k$ to get $(u_k, v_k)$ plus a visibility flag. The query samples image features $F_k(u_k, v_k)$ by bilinear interpolation, averages visible views, and updates itself. The head predicts class plus box $(x, y, z, l, w, h, \theta, v_x, v_y)$ . Training matches predictions to truths one-to-one, so duplicates are penalized directly.

Worked example

Query 42 starts at $(5.0, 8.0, 1.0)$ m. It projects into front-left at $(410, 380)$ visible and front at $(120, 400)$ visible, and four other views invisible. Sampled features are $[0.5, 0.3]$ and $[0.4, 0.2]$ , averaged to $[0.45, 0.25]$ . After layer 3 the point moves to $(5.4, 8.3, 0.9)$ m with car score $0.61$ ; after layer 6 it settles at $(5.5, 8.4, 0.9)$ m, score $0.87$ , yaw $0.10$ rad. The $0.5$ m walk came from image evidence, not anchors.

Watch Out For

Queries stuck in empty space

With 900 queries spread over $100$ m, a query starting $15$ m from any object samples road and never moves. The symptom is low recall at range. Initialize reference points from a coarse heat prior instead of uniform random.

Duplicate boxes across view seams

A car on a view boundary projects into two cameras with slightly different features and two queries claim it. The set loss usually suppresses this, but seam duplicates persist when views disagree by more than $0.5$ m in depth.

The Quick Version

DETR3D keeps 900 3D queries refined over six layers by sampling visible views.
Projection decides which cameras each query listens to per layer.
One-to-one matching removes NMS entirely.
Do not use it when you also need dense occupancy; it outputs sparse boxes only.
Six refinement rounds turn a $0.5$ m initial error into a tight box.