Curiosity-Driven Exploration (ICM)

Why Does This Exist?

In standard reinforcement learning, agents rely on extrinsic reward signals provided by the environment—such as scoring points in a game or crossing a finish line. In complex, sparse-reward environments (such as navigating a maze or solving a multi-stage robotic manipulation task), an extrinsic reward is only awarded after thousands of coordinated actions. An agent executing random actions (such as $\epsilon$-greedy exploration) will virtually never stumble onto the goal, causing training to stall indefinitely.

To overcome sparse feedback, researchers introduced intrinsic motivation: rewarding the agent internally for curiosity and novelty. Early formulations generated intrinsic rewards based on prediction error in raw observation space: if an agent predicts the next pixel frame $\hat{s}_{t+1}$ given state $s_t$ and action $a_t$, the squared pixel error $\| \hat{s}_{t+1} - s_{t+1} \|_2^2$ serves as a curiosity bonus.

However, raw prediction error suffers from a fatal flaw known as the Noisy TV Problem. If the environment contains irrelevant, uncontrollable stochasticity—such as leaves blowing in the wind, water ripples, or a television monitor playing random static noise—the forward dynamics of those pixels are fundamentally unpredictable. An agent rewarded for raw pixel prediction error will walk up to the television and stare at the static indefinitely, mesmerized by infinite, meaningless novelty while never exploring the actual task.

The Intrinsic Curiosity Module (ICM), formulated by Deepak Pathak et al. (2017), solves this problem by projecting raw states into a compact latent feature space $\phi(s)$ learned via inverse dynamics. By forcing the latent features to predict the agent's chosen action, ICM strips away all uncontrollable visual distractors. The forward model then predicts transitions exclusively within this noise-immune feature space, generating intrinsic rewards that compel the agent to explore meaningful, agent-controllable environmental dynamics.

Think of It Like This

How It Actually Works

The Dual-Model Architecture and Noisy TV Resolution

Consider an agent interacting with an environment with state $s_t \in \mathcal{S}$, taking action $a_t \in \mathcal{A}$, and observing next state $s_{t+1}$. The Intrinsic Curiosity Module decomposes exploration into two sub-networks: an Inverse Dynamics Model and a Forward Dynamics Model, mediated by a shared Feature Encoder $\phi(s)$.

1   Raw Observation s_t ────► [ Feature Encoder φ ] ────► φ(s_t)2                                                           │3                                ┌──────────────────────────┴───────────────┐4                                ▼                                          ▼5   Raw Observation s_{t+1} ──► [ Feature Encoder φ ] ──► φ(s_{t+1})    Action a_t6                                │                                          │7                                ▼                                          ▼8                  [ Inverse Dynamics g_inv ]                  [ Forward Dynamics f_fwd ]9                                │                                          │10                                ▼                                          ▼11                         Predicted â_t                        Predicted φ̂(s_{t+1})12                                │                                          │13                                ▼                                          ▼14                   Loss: L_inv(â_t, a_t)                      Error: e = φ̂ - φ(s_{t+1})15              (Forces φ to ignore noise TV)                                │16                                                                           ▼17                                                                  Reward: r_int ∝ ||e||²

1. Feature Encoder $\phi(s)$ and the Inverse Dynamics Model

The feature encoder $\phi(s; \theta_E)$ maps high-dimensional observations (e.g., $84 \times 84 \times 4$ raw video frames) into a compact latent vector $\phi(s) \in \mathbb{R}^d$.

The Inverse Dynamics Model $g_{\text{inv}}(\phi(s_t), \phi(s_{t+1}); \theta_I)$ takes the latent representations of two consecutive states and predicts the action $\hat{a}_t$ that caused the transition:

For continuous actions, the inverse loss is the Mean Squared Error:

For discrete actions, $\mathcal{L}_{\text{inv}}$ is the multi-class categorical cross-entropy over action logits.

The Crucial Invariant: The parameters of the feature encoder $\theta_E$ are updated via gradient descent on $\mathcal{L}_{\text{inv}}$. In order to predict the agent's action $a_t$, $\phi(s)$ is pressured to retain only those features that change as a result of the agent's actions or that directly influence the agent. Uncontrollable environmental patterns (flickering televisions, drifting background clouds, moving wallpaper) contain zero mutual information about $a_t$. Consequently, $\mathcal{L}_{\text{inv}}$ gradients ruthlessly scrub uncontrollable noise from the latent embedding $\phi(s)$, rendering the representation immune to the Noisy TV Problem.

2. The Forward Dynamics Model and Intrinsic Reward

The Forward Dynamics Model $f_{\text{fwd}}(\phi(s_t), a_t; \theta_F)$ predicts the future latent embedding $\hat{\phi}(s_{t+1})$ conditioned on the current embedding $\phi(s_t)$ and the executed action $a_t$:

The forward loss measures how inaccurate the forward model's prediction is:

The Intrinsic Curiosity Reward $r_t^i$ supplied to the reinforcement learning policy is directly proportional to this prediction error:

where $\eta > 0$ is a curiosity scaling hyperparameter.

3. Joint Optimization Objective

The reinforcement learning agent optimizes a policy $\pi_\theta(a \mid s)$ to maximize the combined return $r_t = r_t^e + r_t^i$, where $r_t^e$ is the extrinsic task reward. The overall system jointly minimizes three objectives:

where $\beta \in [0, 1]$ is a weighting hyperparameter (typically $\beta = 0.20$, placing greater weight on learning an accurate inverse embedding). Note that gradients from $\mathcal{L}_{\text{fwd}}$ do not update the policy directly; instead, $\mathcal{L}_{\text{fwd}}$ supplies the scalar reward $r_t^i$ used by the policy gradient algorithm (such as PPO or A2C).

Worked numerical example

Let us trace a single forward-inverse calculation through the Intrinsic Curiosity Module step-by-step with concrete numerical values.

Setup

Consider a 2-dimensional latent feature space ($d = 2$) and a 1-dimensional continuous action ($a_t \in \mathbb{R}$):

• Current latent state: $\phi(s_t) = [0.50, 0.20]$
• True next latent state: $\phi(s_{t+1}) = [0.80, 0.40]$
• Executed action: $a_t = [1.00]$
• Curiosity scale: $\eta = 1.0$
• Model trade-off parameter: $\beta = 0.20$

The sub-networks produce the following forward and inverse predictions:

• Forward model prediction: $\hat{\phi}(s_{t+1}) = [0.60, 0.30]$
• Inverse model action prediction: $\hat{a}_t = [0.90]$

Step 1: Compute Forward Prediction Error Vector

Calculate the difference between the forward model's predicted next latent state and the ground-truth next latent state:

Step 2: Compute Squared Norm Error

Calculate the squared $L_2$ norm of the prediction error vector:

Step 3: Compute Intrinsic Curiosity Reward

Evaluate the scalar reward granted to the RL policy:

The agent receives an intrinsic reward of $+0.0250$ for taking action $a_t$ in state $s_t$, encouraging the policy to repeat actions leading to unfamiliar transition dynamics.

Step 4: Evaluate Inverse Dynamics Loss

Compute the Mean Squared Error between the predicted action $\hat{a}_t = 0.90$ and true action $a_t = 1.00$:

Step 5: Evaluate Forward Dynamics Loss

Compute the forward MSE loss:

Step 6: Compute Composite ICM Optimization Loss

Combine the forward and inverse losses using $\beta = 0.20$:

The total composite loss for the curiosity module is $0.0090$. Backpropagating this loss updates the forward model parameters $\theta_F$, the inverse model parameters $\theta_I$, and the shared encoder parameters $\theta_E$.

Code

The following self-contained Python script implements the Intrinsic Curiosity Module, computes forward and inverse dynamics errors, generates intrinsic curiosity rewards, filters uncontrollable visual noise, and validates the numerical example step-by-step.

1from typing import Dict, Tuple2import numpy as np3
4class IntrinsicCuriosityModule:5    """Intrinsic Curiosity Module (ICM) based on Pathak et al. (ICML 2017).6
7    Overcomes the Noisy TV problem by learning an inverse dynamics embedding8    space that filters uncontrollable environmental noise before evaluating9    forward dynamics prediction error.10    """11
12    def __init__(13        self,14        obs_dim: int,15        feature_dim: int,16        action_dim: int,17        eta: float = 1.0,18        beta: float = 0.2,19    ) -> None:20        self.obs_dim = obs_dim21        self.feature_dim = feature_dim22        self.action_dim = action_dim23        self.eta = eta24        self.beta = beta25
26    def compute_intrinsic_reward(27        self,28        phi_next: np.ndarray,29        hat_phi_next: np.ndarray,30    ) -> Tuple[float, np.ndarray]:31        """Calculates curiosity reward: r_int = (eta / 2) * ||hat_phi(s_{t+1}) - phi(s_{t+1})||_2^2."""32        error_vec = hat_phi_next - phi_next33        sq_norm = float(np.sum(error_vec ** 2))34        r_int = (self.eta / 2.0) * sq_norm35        return r_int, error_vec36
37    def compute_losses(38        self,39        action: np.ndarray,40        hat_action: np.ndarray,41        phi_next: np.ndarray,42        hat_phi_next: np.ndarray,43    ) -> Dict[str, float]:44        """Calculates inverse, forward, and total composite ICM losses."""45        # Inverse loss: MSE on predicted action L_inv = 0.5 * ||hat_a_t - a_t||^246        inv_loss = 0.5 * float(np.sum((hat_action - action) ** 2))47
48        # Forward loss: MSE on predicted latent embedding L_fwd = 0.5 * ||hat_phi - phi||^249        fwd_loss = 0.5 * float(np.sum((hat_phi_next - phi_next) ** 2))50
51        # Combined ICM objective: beta * L_fwd + (1 - beta) * L_inv52        total_loss = self.beta * fwd_loss + (1.0 - self.beta) * inv_loss53
54        return {55            "loss_inv": inv_loss,56            "loss_fwd": fwd_loss,57            "loss_icm": total_loss,58        }59
60# --- Step-by-Step Numerical Example Verification ---61icm = IntrinsicCuriosityModule(obs_dim=4, feature_dim=2, action_dim=1, eta=1.0, beta=0.2)62
63# True latent states and actual action executed64phi_t = np.array([0.50, 0.20], dtype=np.float64)65phi_next = np.array([0.80, 0.40], dtype=np.float64)66action = np.array([1.00], dtype=np.float64)67
68# Network predictions from forward and inverse models69hat_phi_next = np.array([0.60, 0.30], dtype=np.float64)70hat_action = np.array([0.90], dtype=np.float64)71
72# 1. Compute intrinsic reward and prediction error vector73r_int, err_vec = icm.compute_intrinsic_reward(phi_next, hat_phi_next)74
75# 2. Compute composite training losses76losses = icm.compute_losses(action, hat_action, phi_next, hat_phi_next)77
78print(f"Prediction Error Vector e: [{err_vec[0]:.2f}, {err_vec[1]:.2f}]")79# -> Prediction Error Vector e: [-0.20, -0.10]80
81squared_norm = float(np.sum(err_vec ** 2))82print(f"Squared L2 Norm ||e||^2: {squared_norm:.4f}")83# -> Squared L2 Norm ||e||^2: 0.050084
85print(f"Intrinsic Curiosity Reward r_int: {r_int:.4f}")86# -> Intrinsic Curiosity Reward r_int: 0.025087
88print(f"Inverse Dynamics Loss L_inv: {losses['loss_inv']:.4f}")89# -> Inverse Dynamics Loss L_inv: 0.005090
91print(f"Forward Dynamics Loss L_fwd: {losses['loss_fwd']:.4f}")92# -> Forward Dynamics Loss L_fwd: 0.025093
94print(f"Total ICM Composite Loss L_ICM: {losses['loss_icm']:.4f}")95# -> Total ICM Composite Loss L_ICM: 0.009096
97# Assertions matching worked example98np.testing.assert_allclose(err_vec, [-0.20, -0.10], atol=1e-5)99np.testing.assert_allclose(squared_norm, 0.0500, atol=1e-5)100np.testing.assert_allclose(r_int, 0.0250, atol=1e-5)101np.testing.assert_allclose(losses["loss_inv"], 0.0050, atol=1e-5)102np.testing.assert_allclose(losses["loss_fwd"], 0.0250, atol=1e-5)103np.testing.assert_allclose(losses["loss_icm"], 0.0090, atol=1e-5)104print("ICM numerical assertions passed successfully.")105# -> ICM numerical assertions passed successfully.

Watch Out For

The Quick Version

• Solves Sparse-Reward Exploration: ICM generates intrinsic exploration rewards proportional to forward dynamics prediction error, allowing agents to solve complex sparse-reward environments without extrinsic guidance.
• Immune to the Noisy TV Problem: By embedding raw visual states through an inverse dynamics model trained to predict the agent's action, ICM filters out uncontrollable background noise and random distractions.
• Dual-Branch Architecture: An inverse model predicts action $\hat{a}_t$ from $(\phi(s_t), \phi(s_{t+1}))$, while a forward model predicts next embedding $\hat{\phi}(s_{t+1})$ from $(\phi(s_t), a_t)$.
• Self-Disappearing Reward: As the agent visits an environment repeatedly, the forward model masters its dynamics, driving prediction error and intrinsic rewards down to zero and naturally shifting policy focus to extrinsic goals.

What to Read Next

• Exploration and Intrinsic Motivation
• Random Network Distillation (RND)
• Count-Based Exploration