nuScenes Multimodal Driving

City drives with cameras, radar and LiDAR plus 3D boxes and map layers, built for full sensor fusion rather than camera alone.

nuScenes fuses camera, radar and LiDAR with 3D boxes and maps, so bird's-eye perception trains on aligned sensors

Why Does This Exist?

One camera misses depth, occlusions and night cues. Real stacks fuse cameras with radar and LiDAR on a shared map. nuScenes recorded city drives with that full rig plus 3D boxes and map layers, so BEV fusion could be trained and scored on aligned sensors.

This page covers sensor roles, 3D I/O and appropriate use. Point clouds are in point clouds.

Think of It Like This

Three witnesses plus a street map

One witness describes colors, one hears motion through fog, one measures distances, and all share the same street map.

Cameras, radar, LiDAR and maps work that way. No single witness carries the scene, but together they cover blind spots.

How It Actually Works

Purpose is 3D detection, tracking and BEV segmentation with fusion. Inputs are multi-camera images, radar returns, LiDAR sweeps and map layers with calibration and time sync. Outputs are 3D boxes, tracklets and BEV masks. Appropriate use is fusion benchmarking and map-aware perception. Verify official version and split before quoting scene counts.

Toy fusion walkthrough

Take 10 night cars where cameras alone find 6 and radar adds 3 more distinct ones. Fused recall is $(6 + 3) / 10 = 0.9$ versus camera-only $6 / 10 = 0.6$ . The $0.3$ lift is the fusion prize, paid for with calibration and sync work.

Code

Fused recall arithmetic on toy counts.

1def recall(found: int, total: int) -> float:2    return round(found / max(total, 1), 3)3
4
5print(recall(6, 10), recall(9, 10))6# -> 0.6 0.9

Camera-only 0.6 rises to fused 0.9 when radar adds three.

Watch Out For

Ignoring time sync on moving scenes

Sensors stamp at different rates, so naive fusion smears fast objects. The symptom is trailing boxes on buses and bikes. Fix it by interpolating to a shared timestamp and checking moving-object slices.

The Quick Version

nuScenes aligns cameras, radar, LiDAR and maps with 3D boxes for fusion.
Inputs are synced sweeps plus maps, outputs are 3D boxes, tracks and BEV masks.
Toy cameras 6 of 10 give 0.6 recall, fused 9 of 10 give 0.9.
Use it to benchmark map-aware fusion and BEV perception.
Verify version officially and honor time sync.