End-to-End Driving

End-to-end driving maps sensor input straight to steering and speed, learning the whole behavior from demonstrations instead of hand-built modules.

Demonstrations train a direct sensor-to-control map whose small errors compound in closed loop

Why Does This Exist?

Modular stacks need labeled lanes, boxes, and trajectories for every subtask, plus hand-tuned handoffs between them. End-to-end driving asks whether demonstrations alone can teach the map from sensors to control, cutting annotation and letting the network discover its own intermediates. It exists to trade interpretability and guarantees for data scale and simplicity, and to force honest closed-loop evaluation.

For the modular alternative see autonomous driving vision and for tracks see multi-object tracking.

Think of It Like This

Learning to cook by watching, not by recipes

You watch a chef for a hundred hours and copy hand motions without ever reading a recipe. You cook well until the kitchen changes, when no recipe tells you what to adjust. End-to-end models copy driving the same way. The analogy stops at kitchens: the model also inherits every bad habit in the demonstrations.

How It Actually Works

Demonstrations pair sensor frames $X$ with expert actions $a$ such as steering and speed. Imitation training minimizes action error per frame. At deployment the model acts, the world shifts, and the next input comes from its own past actions rather than the expert distribution. Small per-frame errors compound into lane departure within seconds. Evaluation therefore runs closed-loop in simulation or on road, measuring route completion and infractions rather than per-frame loss.

Worked example

A model with 0.02 rad mean steering error per frame at 10 Hz drifts about 0.02 rad times speed each step. At 10 m per s the lateral drift grows near 0.2 m per s before correction, so after 5 s the car sits about 1 m off center without feedback correction. Open-loop validation calls 0.02 rad excellent while closed-loop testing records a curb strike. That gap is why papers report both numbers.

Code

This snippet shows compounding from a tiny bias.

1import numpy as np2
3
4def rollout(bias: float, steps: int = 50, speed: float = 10.0, dt: float = 0.1) -> float:5    lat = 0.06    for _ in range(steps):7        lat += speed * bias * dt  # tiny per-step error integrates8    return round(lat, 2)9
10
11print(rollout(0.02))  # -> 1.012print(rollout(0.005))  # -> 0.25

A 0.02 rad bias becomes a 1 m offset in 5 s. Small open-loop numbers still fail closed-loop.

Watch Out For

Offline metrics that hide drift

Per-frame action error falls while route completion stays flat, because the test set contains only expert states the car never leaves during training. The symptom is a demo that steers smoothly for a minute then departs at the first novel glare. Evaluate closed-loop early, add corrective or perturbed data, and never ship on open-loop loss alone.

The Quick Version

End-to-end maps sensors directly to control from demonstrations.
Annotation drops but interpretability and guarantees drop too.
Per-frame error compounds into trajectory failure within seconds.
Closed-loop route metrics gate releases, not open-loop loss.
Use it for behavior breadth, with safety wrappers outside the network.