Multi-Sensor Fusion for 3D Detection

Lidar plus cameras in one bird's-eye frame keep boxes alive when night, rain, or glare knocks out either sensor alone.

Lidar and camera bird's-eye grids align in one ego frame and detect once together

Why Does This Exist?

Lidar measures depth and fails in spray. Cameras read texture and fail at night. Radar reports speed and localizes poorly. Any single sensor has weather where it quits, and 3D object detection built on one sensor inherits that quit.

Late fusion merges finished boxes and inherits each sensor's misses. Multi-sensor fusion for 3D detection moves the merge earlier: encode lidar and cameras into separate bird's-eye grids $G_L$ and $G_C$ in one shared ego frame at time $t$ , stack them, and run one head. Either branch can then carry the frame when the other degrades.

This page owns the 3D detection merge. The general fusion choice is sensor fusion, the driving system around it is BEV perception and sensor fusion, and the single-sensor baselines are LiDAR based and camera-based. Dense fusion is BEVFusion and sparse refinement is TransFusion, built on heads like CenterPoint and SECOND.

Think of It Like This

Two tracing sheets on one light table

One friend traces building outlines from a laser scan onto tracing paper. Another traces window colors from photos onto a second sheet. You lay both sheets on one light table, align the corner marks, and draw the final map from the stack.

Grids are the sheets and ego calibration is the corner marks. Training with one sheet sometimes missing teaches you to read either alone. The sitting stops at alignment: friends tolerate sloppy corners while grids split one car into two boxes on a $0.4$ degree error.

How It Actually Works

Lidar points voxelize into $G_L$ of shape $180 \times 180 \times 128$ at $c = 0.5$ m, covering $90$ m from $-45$ to $+45$ m. Camera views lift into $G_C$ on the same footprint through depth splatting. Motion compensation shifts both grids into ego coordinates at time $t$ before fusion. The stack $[G_L, G_C]$ has $256$ channels, passes three $3 \times 3$ convolutions, then feeds a center head predicting heat plus $(x, y, z, l, w, h, \theta)$ .

Sparse fusion flips the order: run the lidar head first, keep the top $200$ boxes as queries, then refine each with image cross-attention at its projected corners. Dense fusion serves occupancy too; sparse fusion spends compute on objects only. Both train with sensor dropout, often $30$ percent camera dropout, so the lidar path stays strong when night removes color.

Worked example

Cell $(90, 100)$ covers $x = (90 - 90) \times 0.5 = 0.0$ m and $y = (100 - 90) \times 0.5 = 5.0$ m. The lidar branch gives $[0.7, 0.1]$ from $6$ points on a car door. The camera branch gives $[0.5, 0.6]$ from a headlight patch at predicted depth $5.2$ m. Concatenated $[0.7, 0.1, 0.5, 0.6]$ fuses to $[0.83, 0.52]$ , and the head emits car score $0.90$ with center $(0.1, 5.1, 0.9)$ m. Either branch alone scored below $0.70$ in night rain.

Alignment sets the ceiling. A $0.4$ degree yaw error at $50$ m shifts the camera grid by $50 \times \tan(0.4^{\circ}) \approx 0.35$ m, pushing lidar and camera peaks into adjacent cells so one car becomes two weak boxes. Online calibration plus static-pole checks hold the stack together.

Code

Cell mapping plus the yaw-shift check behind ghost boxes.

1import math2
3
4def cell_90(x: float, y: float, origin: float = -45.0, cell: float = 0.5) -> tuple[int, int]:5    """BEV cell on the 180 grid for an ego point."""6    return int((x - origin) / cell), int((y - origin) / cell)7
8
9def yaw_shift(range_m: float, deg: float = 0.4) -> float:10    """Lateral shift from a calibration yaw error."""11    return round(range_m * math.tan(math.radians(deg)), 2)12
13
14print(cell_90(0.0, 5.0))  # -> (90, 100)15print(yaw_shift(50.0))  # -> 0.35

Watch Out For

Calibration drift that splits one car in two

A $0.4$ degree yaw error moves camera evidence $0.35$ m at $50$ m while lidar stays put. The head sees two half peaks and emits two boxes at $0.45$ each instead of one at $0.90$ . Re-calibrate online, verify on static poles, and alert when pole reprojection exceeds $0.15$ m.

Fusion trained only on clean paired frames

Always-on both sensors teaches the fuser to lean on cameras, so night frames drop $10$ points and the lidar path cannot carry alone. Train with $30$ percent camera dropout plus lidar-only batches, and test each sensor removed before trusting rain gains.

The Quick Version

Multi-sensor 3D fusion stacks lidar and camera bird's-eye grids in one ego frame and detects once.
A $180 \times 180$ grid at $0.5$ m cells aligns both sensors over $90$ m.
Our cell $(90, 100)$ at $(0, 5)$ m fuses to $0.90$ when each branch alone sits below $0.70$ .
A $0.4$ degree yaw error shifts cameras $0.35$ m at $50$ m and splits boxes.
Dropout training keeps either sensor able to carry the frame alone.