Robot Learning: Imitation to RL

1. Contact-Rich Manipulation

Classical robotics excels at pick-and-place in free space where manipulators move without touching unattached objects. However, everyday tasks like pushing a block, threading a cable, or assembling gears require contact-rich interactions.

When a robot pusher contacts a block on a tabletop, physical interaction is governed by non-smooth Coulomb friction, stick-slip transitions, and minute surface deformations. Analytical equations cannot easily model the discontinuous jumps between sticking ($\|f_t\| \le \mu_s f_n$) and sliding ($\|f_t\| = \mu_k f_n$). Rather than deriving intractable contact physics by hand, robot learning trains deep neural policies $\pi_\theta(a \mid s)$ directly from empirical interaction data.

2. Recording Demonstrations via Teleoperation

Imitation learning begins by capturing how human operators solve manipulation problems. Using bilateral leader-follower arms, VR controllers, or optical motion tracking, an expert guides the robot through the task:

1. The follower robot records its sensor state $s_t = (q_t, \dot{q}_t, x_{\text{block}}, I_t)$ at $50\,\text{Hz}$.
2. The user's motion input is logged as the executed action $a_t = (\Delta x, \Delta y, \Delta z, \Delta\text{grip})$.
3. Completing $N = 50$ demonstration trials yields an offline dataset $\mathcal{D} = \{(s_t^{(i)}, a_t^{(i)})\}$ of 2,500 expert transition pairs.

The resulting dataset contains successful manipulation strategies without requiring the engineer to specify intermediate waypoint coordinates.

3. Supervised Behavior Cloning

Behavior Cloning (BC) treats policy learning as supervised regression. A multi-layer perceptron, transformer, or diffusion network parameterized by weights $\theta$ takes the observed state $s_t$ and predicts action $\hat{a}_t = \pi_\theta(s_t)$.

The training objective minimizes mean squared error over demonstration dataset $\mathcal{D}$:

$$\mathcal{L}(\theta) = \frac{1}{|\mathcal{D}|} \sum_{(s, a) \in \mathcal{D}} \|\pi_\theta(s) - a\|_2^2$$

Gradient descent adjusts $\theta$ until policy outputs replicate expert decisions. When evaluated on demonstration replays, the trained network achieves near-zero regression loss.

4. Compounding Covariate Shift

Despite low offline validation loss, deploying a pure behavior cloning policy on physical hardware often fails catastrophically. Supervised learning assumes data samples are independent and identically distributed ($i.i.d.$). In robotics, the policy's actions directly choose future states:

1. At step $t=10$, a minor motor tracking error of $\epsilon = 5^\circ$ nudges the pusher $0.02\,\text{m}$ to the side.
2. At step $t=11$, the robot enters a state $s_{11}$ where the pusher is misaligned with the block. Because the expert drove perfectly, state $s_{11}$ never existed in training dataset $\mathcal{D}$.
3. Faced with an out-of-distribution observation, the neural network extrapolates unpredictably, commanding another erratic action.

Rather than self-correcting, single-step mistakes accumulate quadratically:

$$\text{Total Expected Error} \le \mathcal{O}(\epsilon \cdot T^2)$$

By step $t=25$, lateral tracking error explodes to $0.28\,\text{m}$, leaving the robot pushing empty air while the block sits untouched.

5. Interactive Dataset Aggregation with DAgger

The Dataset Aggregation (DAgger) algorithm resolves covariate drift by teaching the policy how to recover from its own errors:

1. Rollout: Let current policy $\pi_\theta$ execute on hardware or in a simulator, allowing it to drift into unfamiliar states $s'$.
2. Expert Query: Ask the human expert what action $a^*(s')$ should be taken from that specific recovery state.
3. Aggregate: Append these recovery pairs to the dataset: $\mathcal{D} \leftarrow \mathcal{D} \cup \mathcal{D}_{\text{new}}$.
4. Retrain: Retrain $\pi_\theta$ on the enriched dataset.

By populating the training distribution with corrective actions from deviated trajectories, DAgger reduces cumulative error from quadratic $\mathcal{O}(\epsilon T^2)$ down to linear $\mathcal{O}(\epsilon T)$. In practice, adding 200 interactive recovery labels shrinks test tracking error from $0.28\,\text{m}$ down to $0.015\,\text{m}$.

6. The Reinforcement Learning Alternative

When human demonstrations are difficult or expensive to collect, Reinforcement Learning (RL) discovers behaviors through trial and error. The robot acts as an autonomous agent in a Markov Decision Process:

• At each tick $t$, policy $\pi_\theta(a_t \mid s_t)$ samples an action.
• The environment transitions to next state $s_{t+1}$ and emits scalar reward $r_t$.
• For block pushing, the reward penalizes distance to goal: $r_t = -\|x_{\text{block}} - x_{\text{target}}\|_2$.

The optimization objective maximizes expected discounted return across horizon $T$:

$$J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{t=0}^T \gamma^t r_t \right]$$

Actor-critic algorithms (such as PPO or SAC) compute policy gradients $\nabla_\theta J(\theta)$ to guide behavior toward high-reward configurations without requiring any human demonstrations.

7. The Peril of Real-World Exploration

While conceptually elegant, training reinforcement learning directly on physical hardware faces severe practical barriers:

1. Hardware Destruction: Early in training, random exploratory actions ($a \sim \mathcal{N}(0, \sigma^2)$) slam manipulators into tables, strip actuator gearheads, and overstretch joint cables.
2. Sample Inefficiency: Standard model-free RL algorithms require roughly $10^7$ interaction steps to converge. Running $10^7$ steps at $20\,\text{Hz}$ requires 139 hours of continuous motion.
3. Reset Overhead: Every time the block flies off the table, a human engineer must physically fetch it and reset the scene, making unattended overnight training impossible.

8. Massively Parallel Simulation

To bypass physical constraints, modern robot learning shifts training into GPU-accelerated physics engines (such as Isaac Sim or MuJoCo).

By parallelizing rigid-body dynamics on graphics hardware, a single workstation simulates 4,096 robot arms simultaneously. Dropped blocks or joint collisions trigger instant software resets:

$$\text{Throughput} = 4{,}096 \times 60\,\text{FPS} \approx 245{,}760\,\text{steps per second}$$

A full 10-million-step training run completes in under 15 minutes of wall-clock time, with zero broken motors and no manual intervention.

9. Bridging the Sim-to-Real Gap with Domain Randomization

A policy trained in simulation rarely works on real hardware out of the box. This discrepancy is the Sim-to-Real Gap, caused by unmodelled physical effects: actuator latency, gear backlash, camera distortion, and inaccurate friction models.

Domain Randomization solves this by randomizing physics parameters across simulated environments:

• Table friction sampled uniformly: $\mu \sim \mathcal{U}(0.30, 0.80)$.
• Block mass randomized: $m \sim \mathcal{U}(0.40, 0.60)\,\text{kg}$.
• Communication latency jittered: $\Delta t \sim \mathcal{U}(5, 25)\,\text{ms}$.

By exposing the neural policy to an entire distribution of dynamics, the real physical world is treated as just another random variation, enabling robust zero-shot policy transfer to physical hardware.

10. Where It Breaks

Both imitation learning and reinforcement learning can fail in ways that classical control never does:

1. Visual Distribution Shift: If an afternoon sunbeam casts an unfamiliar shadow across the tabletop, vision-based policies experience feature collapse. Because the convolutional activations diverge from training, the robot arm may freeze or oscillate violently.
2. Reward Hacking: In reinforcement learning, the agent optimizes the literal mathematical reward, not the designer's intent. If given a reward for "touching the boundary quickly," the robot might fling the block off the table onto the floor to trigger the sensor rather than carefully pushing it to the target.
3. Contact Instability: Fusing high-frequency neural actions into low-level torque controllers without admittance filtering can trigger violent resonance when contacting stiff steel surfaces.

The Quick Version

• Contact-rich manipulation involves non-smooth frictional transitions that defy pure analytical models.
• Teleoperation captures synchronized state-action trajectories from expert human demonstrations.
• Behavior cloning trains neural networks via supervised regression over demonstration dataset pairs.
• Single-step prediction errors compound quadratically, pushing robots into unfamiliar out-of-distribution states.
• DAgger queries experts on deviated rollout trajectories to aggregate essential recovery behaviors.
• Reinforcement learning eliminates demonstration dependence by maximizing scalar environmental reward signals.
• Hardware trial and error risks broken gearheads and unmanageable sample-collection runtimes.
• GPU physics engines parallelize thousands of simulation environments to collect millions of steps.
• Domain randomization exposes policies to physical parameter variations for robust sim-to-real transfer.
• Visual distribution shifts and reward hacking cause policies to freeze or exploit loopholes.