Inverse Reinforcement Learning

Work backward from expert moves to the hidden reward they imply. You match feature counts, then run normal RL on the recovered goal.

Expert path s1 to s3 gives counts 1.9 and 0.9 against learner 1.0 and 0.9, so weights move from 2.0 to 2.45

Why Does This Exist?

Writing rewards by hand is brittle. For driving you want safety, progress, comfort, and legality, all traded off per second. Weight them wrong and the car either freezes or barges ahead. Hand tuning that balance across weather and cities never ends.

Inverse RL asks a person to drive instead. You watch demos and infer the reward they must have been optimizing. Then you run standard RL on that recovered reward, so the agent can handle streets the expert never showed.

This page covers the inversion loop and the ambiguity fix. Copying moves directly is imitation learning, with its simplest form in behavioral cloning. The entropy principled version has its own page in maximum entropy IRL.

The one idea you need first

An MDP is $(S, A, T, \gamma, R)$ for states, actions, dynamics, discount, and reward. Here $R$ is missing and demos fill the gap. Feature counts $\mu$ summarize where time was spent, discounted by $\gamma$ per step. Match those counts and values match too under linear rewards.

Think of It Like This

Guessing the judging rubric

You watch a cooking judge taste ten dishes and pick winners without seeing the score sheet. One winner is salty and fast, another is mild and slow. You list traits per dish, then find weights that make the winners score highest.

Many rubrics fit. All zeros calls every dish tied, which technically explains the picks and teaches nothing. So you pick the least opinionated rubric that still puts winners on top. That extra modesty is the maximum entropy idea, and it keeps you from inventing preferences the judge never showed.

Where the analogy stops: judges score finished plates, while agents steer through sequences where early turns change later options. Counts must be discounted and dynamics aware, not just tallied.

How It Actually Works

Assume linear reward $R(s) = \theta^T \phi(s)$ . Symbols: $\phi(s)$ is the feature vector for a state, $\theta$ holds the weights you seek. Expert demos $\mathcal{D} = \{\tau_1, \dots, \tau_N\}$ give empirical counts $\mu_E$ . Your current policy $\pi_\theta$ gives $\mu(\pi_\theta)$ . The MaxEnt gradient is their gap:

\nabla_\theta \mathcal{L} = \mu_E - \mu(\pi_\theta)

You step $\theta$ , solve RL for the new $\theta$ , remeasure counts, and repeat. Trajectory odds follow $P(\tau) \propto \exp(R(\tau))$ , so high reward paths draw most probability while every path keeps some share. That soft choice is what resolves the all zeros tie.

Worked example

Three states with $\phi(s_1) = [1.0, 0.0]$ , $\phi(s_2) = [0.0, 1.0]$ , $\phi(s_3) = [1.0, 1.0]$ . Start $\theta = [2.0, -1.0]$ , $\gamma = 0.90$ , step $\alpha = 0.50$ .

Expert runs $\tau_E = (s_1 \to s_3)$ :

\mu_E = [1.0, 0.0] + 0.90 \times [1.0, 1.0] = [1.90, 0.90]

Learner under $\theta$ runs $\tau_\pi = (s_1 \to s_2)$ :

\mu(\pi_\theta) = [1.0, 0.0] + 0.90 \times [0.0, 1.0] = [1.00, 0.90]

Gradient:

\nabla_\theta \mathcal{L} = [1.90, 0.90] - [1.00, 0.90] = [0.90, 0.00]

Update:

\theta_{new} = [2.00, -1.00] + 0.50 \times [0.90, 0.00] = [2.45, -1.00]

First weight rises by $0.45$ while the second holds. Next RL round values feature one more, so the learner favors $s_3$ over $s_2$ and edges toward the demo path. Those four vectors are what the diagram plots.

Code

This snippet recomputes the count match with plain lists.

1"""Recompute IRL feature counts and the theta step."""2
3gamma = 0.94alpha = 0.55theta = [2.0, -1.0]6phi = {"s1": [1.0, 0.0], "s2": [0.0, 1.0], "s3": [1.0, 1.0]}7
8mu_e = [phi["s1"][0] + gamma * phi["s3"][0], phi["s1"][1] + gamma * phi["s3"][1]]9mu_pi = [phi["s1"][0] + gamma * phi["s2"][0], phi["s1"][1] + gamma * phi["s2"][1]]10grad = [mu_e[0] - mu_pi[0], mu_e[1] - mu_pi[1]]11theta_new = [theta[0] + alpha * grad[0], theta[1] + alpha * grad[1]]12
13print([round(v, 2) for v in mu_e])14# -> [1.9, 0.9]15print([round(v, 2) for v in mu_pi])16# -> [1.0, 0.9]17print([round(v, 2) for v in grad])18# -> [0.9, 0.0]19print([round(v, 2) for v in theta_new])20# -> [2.45, -1.0]

Watch Out For

Claiming you found the true reward

Many rewards explain the same demos, including zero everywhere. The symptom is treating recovered weights as ground truth and shipping them to new tasks where they steer oddly.

Report the reward as one consistent explanation, not the truth. Prefer MaxEnt or margin guards, test the resulting policy rather than the weights, and expect retraining when features or dynamics shift.

Ignoring the inner RL cost

Each outer step solves RL again to get $\mu(\pi_\theta)$ . The symptom is a prototype that works on three states but stalls on larger worlds because the inner loop dominates runtime.

Budget that loop first. Use warm starts, short rollouts early, or adversarial shortcuts that skip full planning. If demos are clean and coverage is narrow, direct cloning often wins on cost.

The Quick Version

Inverse RL recovers reward weights $\theta$ from expert feature counts $\mu_E$ .
Matching counts matches values under linear $R(s) = \theta^T \phi(s)$ .
Here expert $[1.90, 0.90]$ minus learner $[1.00, 0.90]$ gives gradient $[0.90, 0.00]$ .
Update moves $\theta$ from $[2.00, -1.00]$ to $[2.45, -1.00]$ , favoring the demo path.
Rewards stay ambiguous, so use MaxEnt guards and judge the learned policy, not the weights.