Camera-Based 3D Detection

Cameras alone can place metric 3D boxes by lifting views into bird's-eye space, trading lidar precision for cost and density.

Camera views lift into one bird's-eye grid that emits metric 3D boxes without lidar

Why Does This Exist?

Driving planners need boxes in meters with heading, not flat rectangles. A 2D box cannot tell a truck beside you from a billboard ahead, which is why 3D object detection outputs $(x, y, z, l, w, h, \theta)$ cuboids. Lidar measures those cuboids directly but costs more than the camera rig and thins fast with range.

Cameras are cheap, dense, and blind to scale. One pixel could be a toy car near or a real car far. Stereo vision recovers scale from overlap, yet surround cameras point outward with almost no shared view. Camera-based 3D detection exists to close that gap: lift all views into one bird's-eye grid in meters, then detect there with the same heads lidar uses.

This page owns the modality choice, when cameras suffice and where they fail. The dense attention route is BEVFormer, the explicit depth route is BEVDet, and the sparse query route is DETR3D. The laser alternative is LiDAR based, and the combination of both is multi-sensor fusion. Camera geometry basics are image formation and camera model.

Think of It Like This

Six balconies and one courtyard map

A hotel courtyard is ringed by six balconies, each with a photographer shooting straight out. No two photos overlap, so no stereo pair exists. You stand below with a blank courtyard map marked in meters.

For each square of the map you ask only the photographers whose view contains that square what they see there, then you pencil the square in. A square at the fountain gets detailed reports from two balconies; a far corner gets vague ones. After all squares, the map holds boxes for every chair and cart in meters, drawn from flat photos alone.

Map squares are bird's-eye cells, photographers are cameras, and asking is projection plus sampling. The sitting stops at depth certainty: balcony reports guess distance while a laser would measure it, so far squares stay fuzzy.

How It Actually Works

Each camera has a projection matrix $P$ that maps an ego point $(x, y, z)$ in meters to a pixel $(u, v)$ plus a visibility flag. $P$ combines intrinsics (focal length, center) with extrinsics (camera pose on the car). Lifting inverts this direction: given pixels, decide which bird's-eye cell $(i, j)$ each pixel supports. The grid $G$ has $H = 200$ by $W = 200$ cells at $c = 0.5$ m, covering $100$ m from $-50$ to $+50$ m in both axes.

Two families lift differently. Explicit methods predict a depth distribution $D(u, v)$ over bins and splat $D_b \cdot F(u, v)$ along the ray into $G$ . Attention methods keep $G$ as queries and let each cell sample image features through deformable cross-attention at its projected references. Sparse methods skip $G$ and refine $900$ object queries in 3D directly. All three end in the same head: class heat plus box $(x, y, z, l, w, h, \theta)$ with velocity.

Worked example

Cell $(100, 120)$ means $x = (100 - 100) \times 0.5 = 0.0$ m and $y = (120 - 100) \times 0.5 = 10.0$ m in ego coordinates. A front pixel at $(800, 450)$ carries feature $[0.4, 0.2]$ with depth weights $[0.05, 0.70, 0.20, 0.05]$ over bins $\{9, 10, 11, 12\}$ m, summing to $1.0$ .

The $10$ m bin dominates: $0.70 \times [0.4, 0.2] = [0.28, 0.14]$ lands in cell $(100, 120)$ . Eleven such pixels pool to about $[3.08, 1.54]$ , and the head emits car score $0.79$ with center $(0.1, 10.1, 0.9)$ m and yaw $0.02$ rad. The $0.1$ m center error passes easily while a $30$ degree yaw error would fail, which is why heading supervision matters more than sub-decimeter centers at this range.

Code

Grid math plus depth-weighted lifting for one pixel.

1def cell_from_ego(x: float, y: float, origin: float = -50.0, cell: float = 0.5) -> tuple[int, int]:2    """BEV cell indices for an ego point."""3    return int((x - origin) / cell), int((y - origin) / cell)4
5
6def lift(feature: list[float], w: float) -> list[float]:7    """Weight one feature by its depth probability."""8    return [round(v * w, 3) for v in feature]9
10
11print(cell_from_ego(0.0, 10.0))  # -> (100, 120)12print(lift([0.4, 0.2], 0.70))  # -> [0.28, 0.14]

Watch Out For

Depth error that grows with range squared

A 1 m depth mistake at $10$ m shifts the box $1$ m, but the same angular mistake at $50$ m shifts it $5$ m. The symptom is clean near-field scores with far-field duplicates and splits. Supervise depth where lidar exists, use finer bins near and coarser far, and gate far boxes at lower score until tracking confirms them.

Ego vehicle pixels splatted as obstacles

Hood and roof pixels visible in every view splat into center cells and fake a permanent obstacle at $(0, 0)$ m. The symptom is a confident box that never moves with the world. Mask the ego region in all views before lifting and verify center cells stay empty on parked logs.

The Quick Version

Camera-based 3D detection lifts surround views into a metric bird's-eye grid, then detects there.
A $200 \times 200$ grid at $0.5$ m cells covers $100$ m without any lidar point.
Explicit depth, attention queries, and sparse queries are three lifting choices with the same head.
Our pixel puts $0.70$ weight on $10$ m, contributing $[0.28, 0.14]$ to cell $(100, 120)$ .
Range magnifies depth mistakes, so supervise depth and mask the ego vehicle first.