Imitation Learning from Video

The internet holds millions of demonstrations with no action labels. Imitation from video learns skills by watching: inferring what the demonstrator did from how the pixels changed.

An inverse model labels frame pairs with actions at 0.75 accuracy, and the policy inherits every mislabel

Why Does This Exist?

Action labels are scarce and videos are everywhere. Robots log state-action pairs by the thousand; humans upload billions of demonstration videos with no joint angles attached. Behavior cloning cannot touch that bounty — it needs the actions — while reinforcement learning pays millions of trial frames per skill. The gap between abundant watching and expensive doing is where video imitation lives.

Imitation from video infers the missing actions: an inverse dynamics model reads consecutive frames and guesses the action between them, producing pseudo-labels that behavior cloning then distills into a policy. The trial-and-error sibling is RL from pixels; the futures engine is video prediction.

The one idea you need first. An inverse dynamics model maps frame pairs to actions: $g(o_t, o_{t+1}) \to \hat{a}_t$ . It trains where actions are known (the robot's own logs, simulation), then labels videos where they are not. The embodiment gap is the catch: human hands in the video differ from robot grippers executing, so inferred actions need alignment across bodies.

Think of It Like This

Learning a dance from silent video

No instructor counts beats; you watch a dancer on mute, infer weight shifts from hip motion, and rehearse before a mirror. The mirror matters: your body differs from hers, so copied angles become adapted ones — same rhythm, different limbs.

Video imitation is that rehearsal formalized. The inverse model is your eye inferring shifts from frames; behavior cloning is the muscle memory; the mirror is embodiment alignment translating human motion into gripper motion. Skip the mirror and the dance fits a body you do not have.

How It Actually Works

Two stages run in sequence. First, train the inverse model on action-labeled trajectories (robot play data, simulators with known controls) to predict actions from frame pairs, then run it over unlabeled demonstration video to emit pseudo-action labels. Second, behavior-clone a policy on the pseudo-labeled frames. Alignment tricks bridge embodiments: viewpoint-invariant features, hand-to-gripper retargeting, and cycle-consistency losses that keep translated motions faithful.

Quality compounds downward: inverse-model errors become policy training data, and policy errors compound at rollout. Both stages need independent validation.

Worked example

Four frame pairs from a reaching video, inverse-model predictions versus hand-checked truth:

Pair 1: predicted up, true up — correct.
Pair 2: predicted up, true up — correct.
Pair 3: predicted left, true left — correct.
Pair 4: predicted up, true right — wrong (a diagonal move read as vertical).

Accuracy $= 3/4 = 0.75$ . The cloned policy trains on these four labels, so one quarter of its supervision pulls rightward reaches upward. If that error concentrates at a bottleneck (the grasp moment), task success can collapse far below $0.75$ — error position matters more than error rate.

Code

The point of this snippet is the inheritance: pseudo-label accuracy caps the policy's supervision quality.

1predicted = ["up", "up", "left", "up"]2truth = ["up", "up", "left", "right"]3
4
5def accuracy(pred: list, tgt: list) -> float:6    hits = sum(1 for p, t in zip(pred, tgt) if p == t)7    return hits / len(tgt)  # fraction of usable supervision8
9
10print(accuracy(predicted, truth))  # -> 0.75

Watch Out For

The embodiment gap eats transferred skills

Human demonstrations show five-fingered hands, compliant wrists, and head-height cameras; the robot has a parallel gripper and a chest camera. Policies cloned across that gap reach for handles with the wrong geometry and grasp at air where fingers would comply.

Retarget explicitly: map human poses to robot kinematics before cloning, or learn embodiment-invariant features with paired multi-body data. Unaligned transfer succeeds only where bodies accidentally agree.

Compounding from pseudo-label noise

Behavior cloning already compounds small errors into drift; training on $75\%$ -correct pseudo-labels starts the drift from a deficit. A policy that would score $0.9$ on true labels can score $0.5$ on inferred ones when errors cluster at bottlenecks.

Validate the inverse model per state region, not just on average, and weight or filter pseudo-labels by confidence. Average accuracy hides the grasp-moment errors that decide tasks.

The Quick Version

Video imitation infers missing actions with inverse dynamics, then clones policies on the pseudo-labels.
The worked fixture gives $3/4 = 0.75$ labeling accuracy, inherited wholesale by the policy.
Human-to-robot transfer needs explicit retargeting; unaligned skills grasp at air.
Pseudo-label noise compounds at rollout, so validate per region with emphasis on bottlenecks.
Demonstrations are abundant but action-free; the inverse model is the bridge that prices them.