TransFusion LiDAR Camera Fusion

TransFusion starts from LiDAR boxes and uses image attention only to refine the boxes that LiDAR already proposed.

Why Does This Exist?

Dense BEV fusion helps but fuses everywhere, including empty road. LiDAR already finds most boxes; its weakness is class and size detail that cameras carry. TransFusion (2022) flips the order: run a LiDAR detector first, keep the top $200$ boxes as queries, then refine each query with image cross-attention. Computation lands on objects, not asphalt.

This page covers the two-stage query refinement. For dense fusion that also serves occupancy, see BEVFusion.

Think of It Like This

A surveyor plus a photographer

A surveyor with a laser marks 200 candidate trees with flags and rough girths. A photographer then walks only to flagged trees, takes close-ups, and corrects the species and girth on each flag. No flag, no photo trip.

LiDAR is the surveyor and cameras are the photographer. The analogy stops at softness: attention weights blend many pixels, not one close-up.

How It Actually Works

Stage one runs a LiDAR head (for example CenterPoint) and keeps queries $Q$ of $200$ boxes with features $f_q$ . Stage two has two transformer layers. Each query projects its 3D corners into the camera views, samples image features at those projections with deformable attention, and updates $f_q$ . The final head re-scores class and regresses residual $(\Delta x, \Delta y, \Delta z, \Delta l, \Delta w, \Delta h, \Delta\theta)$ .

Worked example

A LiDAR query proposes a van at $(8.0, 2.0, 1.0)$ m, size $(4.8, 2.0, 2.1)$ m, score $0.66$ , class truck. Its corners project into the front camera around $(900, 420)$ to $(980, 500)$ . Sampled image features average $[0.6, 0.5]$ , shifting the query feature from $[0.3, 0.2]$ to $[0.55, 0.42]$ . The refined head outputs class van at $0.88$ with center $(8.1, 2.0, 1.0)$ m. The $0.22$ score gain came from the logo pixels LiDAR never saw.

Watch Out For

LiDAR miss never recovered by cameras

If stage one drops a black car at $60$ m, no query exists and image attention never runs there. The symptom is fusion that matches LiDAR-only recall. Keep stage-one threshold low at $0.15$ and let stage two filter.

Image attention on occluded corners

A query corner hidden behind a bus still samples bus pixels and corrupts the update. Gate sampling by a visibility check from the LiDAR depth before attending.

The Quick Version

TransFusion proposes with LiDAR first, then refines 200 queries with images.
Cameras fix class and size; LiDAR fixes position and recall base.
Two transformer layers keep cost on objects, not the whole grid.
Do not use it when LiDAR is absent; it has no camera-only fallback path.
A $0.22$ score lift on vans is typical when logos resolve the class.