Imitation Learning

Learn by copying an expert instead of exploring from scratch. You turn demos into labeled pairs and train a policy the way you train a classifier.

Behavioral cloning turns one steering demo into a supervised update, moving the expert logit from 2.5 to 2.64 while loss sits at 0.32

Why Does This Exist?

Trial and error is a poor teacher when mistakes cost metal. A robot arm that learns by flailing breaks hardware. A car that learns by random steering leaves the lane. Rewards are also thin out there, so pure exploration can take millions of steps to stumble onto anything useful.

Imitation learning skips the search. You record an expert, a human driver or a slow planner that already solves the task, and you copy them. The state $s$ is what the expert saw. The action $a$ is what the expert did. Your job is to fit a policy $\pi(a \mid s)$ that picks the same action in the same state.

This page covers the copy step, called behavioral cloning, plus why small errors snowball and how interactive fixes help. If you want to infer the hidden reward behind the demos instead of copying moves, read inverse RL. If you want the interactive repair in full, read DAgger. Adversarial matching lives in GAIL.

The one idea you need first

Supervised classification maps inputs to labels with a loss like cross entropy. Behavioral cloning does the same trick: input is the state, label is the expert action. If you can train a classifier, you already know the training loop here.

Think of It Like This

Tracing a chef knife cut

You want to chop like a chef. You do not invent the grip by random wrist twists. You watch their hand, place your hand the same way, and repeat the same stroke on the same onion.

That works until your knife drifts half an inch off their line. Now you are cutting air where their onion was, and your next stroke starts from a spot they never showed you. One small offset puts you off the demo path, and every later cut inherits the offset.

Where the analogy stops: a chef can grab your wrist and correct you mid stroke. A cloned policy cannot ask for help unless you build that loop, which is what DAgger adds.

How It Actually Works

Collect demos $\mathcal{D} = \{(s_i, a_i^*)\}$ where $a_i^*$ is the expert action in state $s_i$ . Fit parameters $\theta$ of a policy $\pi_\theta$ by minimizing supervised loss. For discrete actions that is negative log likelihood:

\mathcal{L}(\theta) = \mathbb{E}_{(s, a^*) \sim \mathcal{D}}[-\log \pi_\theta(a^* \mid s)]

In plain words: push probability toward whatever the expert picked. Symbols: $s$ is the observation, $a^*$ is the expert label, $\pi_\theta(a^* \mid s)$ is your predicted probability for that label.

Worked example

One steering demo. State $s = [0.8, -0.4]$ . Expert picks Straight, label $y = [0, 1, 0]$ over actions Left, Straight, Right. Your net outputs logits $z = [1.2, 2.5, 0.3]$ .

Exponentiate: $e^{1.2} = 3.3201$ , $e^{2.5} = 12.1825$ , $e^{0.3} = 1.3499$ . Sum to $16.8525$ . Divide:

\pi = [0.1970, 0.7229, 0.0801]

Loss on the expert move:

\mathcal{L} = -\log(0.7229) = 0.3245

Gradient with respect to logits is prediction minus label, $\pi - y$ :

\nabla_z \mathcal{L} = [0.1970, -0.2771, 0.0801]

With step size $\eta = 0.5$ , the expert logit climbs:

z_2^{new} = 2.5 - 0.5 \times (-0.2771) = 2.6386

So one supervised step nudges Straight from 2.5 to 2.64 and pushes the other two down. That is the whole update the diagram draws.

The catch is deployment drift. With per step error $\epsilon = 0.10$ over $T = 20$ steps, cloning regret can grow quadratically:

\mathbb{E}[\text{cost}] \le \epsilon T^2 = 0.10 \times 400 = 40.0

Interactive methods that label the states you actually visit cut that growth back toward linear.

Code

This snippet recomputes the worked numbers with plain Python so you can check the arithmetic yourself.

1"""Recompute the steering cloning step from the worked example."""2
3import math4
5logits = [1.2, 2.5, 0.3]6exps = [math.exp(z) for z in logits]7total = sum(exps)8probs = [e / total for e in exps]9
10print([round(p, 4) for p in probs])11# -> [0.197, 0.7229, 0.0801]12print(round(-math.log(probs[1]), 4))13# -> 0.324514
15grad = [probs[0] - 0.0, probs[1] - 1.0, probs[2] - 0.0]16print([round(g, 4) for g in grad])17# -> [0.197, -0.2771, 0.0801]18print(round(2.5 - 0.5 * grad[1], 4))19# -> 2.638620print(round(0.10 * 20 * 20, 1))21# -> 40.0

Watch Out For

Treating demos as perfect truth

Cloning copies flaws too. If the expert hesitates at intersections or favors the right lane for comfort, your policy learns the hesitation and the bias as if they were skill. The symptom is a policy that matches validation loss yet drives oddly in the same spots the human drove oddly.

Fix it at the data layer. Filter demos to clean runs, balance the tricky states, and keep a held out route that tests behavior, not just loss. If the expert ceiling is too low, you need reward inference or extra RL, not more cloning.

Copying the wrong cue

Cloning chases correlation. A driving net can learn brake lights on the dashboard reflection instead of the car ahead, because both lined up with braking in the logs. The symptom is sharp validation accuracy plus strange failures when the spurious cue shifts.

Fix it by checking what the policy looks at. Vary lighting, hide the dashboard, and test on a route where the true cause and the false cue disagree. Add those counter cases to training or move to a method that models intent.

The Quick Version

Imitation learning turns expert demos into supervised pairs of state plus expert action.
Behavioral cloning minimizes negative log likelihood on those pairs, with no reward and no Bellman update.
One steering step in this page moves the expert logit from 2.5 to 2.64 at loss 0.3245.
Small slips compound, with cost up to order $\epsilon T^2$ , here $40.0$ for $\epsilon = 0.10$ and $T = 20$ .
Cloning cannot beat the expert and it copies bad habits, so curate demos and test behavior on held out routes.