Autonomous Driving Vision
Self-driving vision turns camera, lidar, and radar into a live map of road, actors, and intent, then hands that map to planning under strict latency.
Why Does This Exist?
A car must answer three questions thirty times per second: what is around me, what will it do next, and what should I do. A missed lane line, a late pedestrian intent, or a 200 ms stall each cause different failures. Autonomous driving vision exists as the umbrella that binds those answers into one pipeline with a shared world model, a fixed latency budget, and a defined operating domain. Without the umbrella, teams optimize detectors in isolation and the car still drives badly.
This page keeps only the residual: the stack, the budgets, and how the pieces fit. Mechanisms live in the children: perception pipeline, lane and sign pages, drivable area, trajectory, BEV, fusion, end-to-end, and occupancy pages linked below. For tracking see multi-object tracking.
Think of It Like This
A rally team, not one driver
The navigator reads the road book, the spotter calls the next bend, and the driver commits. Perception is the navigator, prediction is the spotter, planning is the driver. Lap time comes from handoffs, not one hero. The analogy stops at people: the car also logs every handoff for replay when something goes wrong.
How It Actually Works
Sensors feed a shared world model in ego coordinates. Perception detects lanes, signs, drivable space, and actors. Prediction forecasts actor trajectories over the next few seconds. Planning selects a safe comfortable path subject to traffic rules. The whole chain runs inside a latency budget, often under 100 ms from photon to command, inside an operational design domain that names weather, roads, and speeds the stack is allowed to face.
Worked example
Take a 30 Hz camera. A budget of 100 ms allows about 3 frames of slack end to end. Suppose perception takes 40 ms, prediction 20 ms, and planning 20 ms, leaving 20 ms for middleware and actuation. A detector that improves accuracy by 2 points but adds 50 ms breaks the budget and forces the planner to act on stale tracks, which field tests punish more than the accuracy gain rewards. That arithmetic is why production stacks prune models that win benchmarks.
Watch Out For
Optimizing modules instead of the drive
A detector with higher mAP can still drive worse when its extra latency stales every downstream forecast. The symptom is clean offline metrics with harsh braking on road. Evaluate the closed loop with prediction and planning in place, and keep latency beside accuracy in every report.
The Quick Version
- Autonomous vision chains perception, prediction, and planning on a shared world model.
- Latency and operating domain constrain every module choice.
- This page is the map; child pages own each mechanism.
- Track handoffs and staleness, not just per-module scores.
- Design for the closed loop, not the leaderboard.