Perception Pipeline

The perception pipeline cleans raw sensor data, finds road users and lanes, and tracks them over time into one consistent world model.

Raw sensors pass through calibration and detection into tracking that keeps stable IDs in one world model

Why Does This Exist?

A detector finds boxes in single frames. Driving needs stable identities across frames: the same pedestrian at 10 m, 8 m, and 6 m, with velocity, not three unrelated boxes. The perception pipeline exists to produce that continuity from messy inputs: cameras and lidar that disagree on timing, calibration that drifts, and detections that flicker. It turns per-frame evidence into a tracked world the planner can trust.

For single-frame boxes see object detection and for identity metrics see multi-object tracking.

Think of It Like This

An air-traffic tower

Radar blips arrive noisy and unlabeled. Controllers correlate blips across sweeps, assign call signs, and keep one strip per aircraft with speed and heading. Perception does the same for cars and pedestrians: detect the blip, keep the strip, hand the tower a stable board. The analogy stops at voices: cars never announce themselves, so IDs must survive purely on appearance and motion.

How It Actually Works

Sensors are time-synchronized and extrinsically calibrated into ego coordinates. Each modality runs detection in its native view, then detections lift into 3D and fuse. Data association matches new detections to existing tracks by IoU plus appearance distance, commonly with a Hungarian match. A motion filter such as a Kalman update smooths position and velocity per track. Unmatched detections spawn tracks, unmatched tracks coast briefly then die. The output is a set of tracks with IDs, boxes, velocities, and confidences.

Worked example

A pedestrian detected at lateral positions 5.0 m, 4.7 m, and 4.4 m across three frames at 10 Hz spans 0.2 s with displacement 0.6 m, giving speed 3.0 m per s toward the road. Association matches the three boxes with IoU 0.55 and 0.60 plus close appearance embeddings, keeping ID 7 alive. A fourth frame with no detection coasts the track to 4.1 m with growing uncertainty. Two misses in a row kill it rather than hallucinating a crossing.

Code

This snippet matches two tracks to three detections by IoU.

1import numpy as np2
3
4def iou(a: tuple[float, float, float, float], b: tuple[float, float, float, float]) -> float:5    ax0, ay0, ax1, ay1 = (float(v) for v in a)6    bx0, by0, bx1, by1 = (float(v) for v in b)7    inter = max(0.0, min(ax1, bx1) - max(ax0, bx0)) * max(0.0, min(ay1, by1) - max(ay0, by0))8    ua = (ax1 - ax0) * (ay1 - ay0) + (bx1 - bx0) * (by1 - by0) - inter9    return inter / (ua + 1e-8)10
11
12dets = [(0, 0, 10, 10), (20, 0, 30, 10), (0, 0, 9, 9)]13tracks = [(0, 0, 10, 10), (20, 0, 30, 10)]14m = np.array([[iou(d, t) for t in tracks] for d in dets])15print(np.round(m, 2).tolist())  # -> [[1.0, 0.0], [0.0, 1.0], [0.81, 0.0]]

The third detection matches the first track at 0.81, so ID persists. The Hungarian step would formalize the assignment.

Watch Out For

Calibration drift that splits every track

A two-degree camera shift moves distant boxes by meters while lidar stays put, so fusion sees two disagreeing observations and alternates IDs. The symptom is ID switches clustered right after a bump or heat soak. Monitor extrinsic residuals continuously and re-estimate online instead of trusting factory calibration.

The Quick Version

Perception converts synchronized detections into stable tracked identities.
Association plus motion filtering gives velocity, not just boxes.
Calibration and timing decide track life more than detector choice.
Coast briefly on misses, then kill to avoid ghost pedestrians.
The world model is the product, not the per-frame boxes.