BEV Perception

BEV perception lifts camera views into a top-down grid around the car, where distances and lanes stay true for planning.

Camera features lift along rays into a top-down ego grid that planning can measure

Why Does This Exist?

Perspective images shrink distance and converge parallel lanes, which is the wrong geometry for measuring gaps. Planning wants meters ahead and meters left, stable across cameras. BEV perception exists to provide that frame: every pixel votes into a top-down grid fixed to the car, so a lane 30 m ahead occupies the same cell size as one 10 m ahead. The combined camera-lidar system is taught in BEV perception and sensor fusion; this page owns the camera-to-BEV lift alone.

For boxes in perspective see object detection.

Think of It Like This

Pouring photos onto a floor map

You print six photos around a table map and pour each one outward along sight lines until it lands on the floor grid. Overlapping pours stack where views agree. The grid, not the photos, is what you measure. The analogy stops at paper: the network learns depth weights for the pour instead of using fixed geometry.

How It Actually Works

Each camera has intrinsics $K$ and ego pose $T$ . A backbone extracts features $F$ at stride 8 or 16. A depth head predicts per-pixel depth bins, for example 40 bins from 2 m to 60 m. Features splat along rays into voxels, then collapse along height into a BEV grid such as $200 \times 200$ cells at 0.5 m resolution covering $100 \times 100$ m. A BEV encoder cleans the grid before task heads read detection and segmentation from it.

Worked example

A lane marking at 30 m falls on a pixel whose depth distribution peaks at bin 28 to 30 m with 0.60 mass. Its feature, scaled by 0.60, lands in the BEV cell at 30 m ahead. A second camera sees the same marking with 0.50 mass in the same cell, summing to 1.10 evidence. Thresholding that cell at 0.8 keeps the lane while a single-view threshold of 0.7 would have dropped the 0.60 view alone. Fusion in the grid is the win.

Code

This snippet splats one feature into depth bins.

1import numpy as np2
3
4def splat(feat: float, depth: np.ndarray) -> np.ndarray:5    return feat * depth6
7
8d = np.array([0.1, 0.1, 0.6, 0.2])9print(np.round(splat(2.0, d), 2).tolist())  # -> [0.2, 0.2, 1.2, 0.4]10print(round(float(splat(2.0, d).sum()), 2))  # -> 2.0

Mass is conserved across bins. The peak bin carries the measurement.

Watch Out For

Depth collapse on textureless roads

Blank asphalt gives no stereo or monocular cue, so depth spreads flat and features smear across 10 m of grid. The symptom is wavy lanes on open highway in noon sun. The fix supervises depth where lidar exists, sharpens distributions with a sparsity term, and trusts temporal accumulation over single frames.

The Quick Version

BEV is a top-down ego grid in meters, not pixels.
Cameras lift into it along rays weighted by predicted depth.
Grid cells accumulate multi-view evidence before decisions.
Resolution and range trade detail against memory directly.
This page is the lift; the fused system page adds lidar and radar.