BEVFusion LiDAR Camera Fusion

BEVFusion encodes LiDAR and cameras in separate bird's-eye grids and fuses them before the detection head runs.

Why Does This Exist?

LiDAR gives exact depth but sparse color; cameras give dense color but weak depth. Late fusion merges finished boxes and inherits each sensor's misses. BEV perception and sensor fusion describes the goal, and BEVFusion (2022) gives the mechanism: encode each sensor into its own BEV grid in a shared ego frame, concatenate, and detect once.

This page covers decoupled BEV encoding and fusion. Single-sensor baselines are SECOND for LiDAR and BEVDet for cameras.

Think of It Like This

Two tracing sheets on one light table

One friend traces building outlines from a laser scan onto tracing paper. Another traces window colors from photos onto a second sheet. You lay both sheets on one light table, align the corner marks, and draw the final map from the stack.

BEV grids are the sheets and ego calibration is the corner marks. The analogy stops at dropout: you can train with one sheet missing, which friends cannot do.

How It Actually Works

LiDAR points voxelize into BEV grid $G_L$ of shape $180 \times 180 \times 128$ . Camera views lift into BEV grid $G_C$ of the same footprint through depth splatting. Both grids share ego coordinates at time $t$ after motion compensation. Fusion concatenates $[G_L, G_C]$ to $256$ channels, applies three $3 \times 3$ convolutions, and feeds a center head predicting heat plus $(x, y, z, l, w, h, \theta)$ .

Worked example

Cell $(90, 100)$ covers $(0, 5)$ m. The LiDAR branch gives $[0.7, 0.1]$ from 6 points on a car door. The camera branch gives $[0.5, 0.6]$ from a headlight patch at predicted depth $5.2$ m. Concatenated $[0.7, 0.1, 0.5, 0.6]$ fuses to $[0.83, 0.52]$ , and the head emits car score $0.90$ with center $(0.1, 5.1, 0.9)$ m. Either branch alone scored below $0.70$ in night rain.

Watch Out For

Calibration drift splitting one car in two

A $0.4$ degree yaw error shifts the camera grid $0.35$ m at $50$ m, so LiDAR and camera peaks for one car land in adjacent cells. The head then emits two weak boxes. Re-calibrate online and verify alignment on static poles first.

Camera dropout at night untrained

Training always with both sensors makes the fuser lean on cameras, so night frames drop $10$ points. Train with random camera dropout at $30$ percent so the LiDAR path stays strong.

The Quick Version

BEVFusion encodes each sensor into its own bird's-eye grid, then fuses.
Shared ego coordinates make the grids stackable across sensors and time.
One head detects from the fused grid, which helps night and rain most.
Do not use it when calibration is unknown; misalignment creates ghost boxes.
Separate encoders let either sensor carry the frame when the other fails.