Advanced RL Paradigms

Why Does This Exist?

Standard single-agent reinforcement learning frames intelligence as an agent mapping states to primitive motor actions via a single flat policy in a stationary Markov Decision Process. While effective for video games with dense rewards, this baseline framing collapses when exposed to four fundamental real-world barriers:

1. Extreme Temporal Horizons & Sparse Rewards: If a reward is only awarded after 100,000 steps (such as assembling an engine or solving Montezuma's Revenge), random exploration has virtually zero probability of stumbling onto success.
2. Non-Stationary Environments: In multi-agent ecosystems (warehouse robotics, autonomous fleets), other agents adapt their policies simultaneously, violating the fundamental MDP assumption that transition dynamics are fixed.
3. Exploration Inefficiencies: In environments with no extrinsic reward signals, flat agents remain paralyzed or twitch aimlessly in place.
4. Task Generalization Bottlenecks: A standard RL policy trained to navigate a maze must restart learning from scratch when walls shift slightly.

Advanced RL paradigms expand the formulation of the problem itself. By introducing temporal hierarchy, multi-agent coordination frameworks, self-supervised curiosity rewards, and meta-learning, these architectures scale RL from toy simulations to complex open-ended systems.

Think of It Like This

How It Actually Works

Hierarchies, Intrinsic Curiosity, and Multi-Agent Coordination

Advanced RL decomposes monolithic control into four specialized structural frameworks:

1. Hierarchical RL & The Options Framework (Sutton, Precup & Singh, 1999)

An option $\omega \in \Omega$ represents a temporally extended macro-action defined by a 3-tuple $\langle \mathcal{I}_\omega, \pi_\omega, \beta_\omega \rangle$:

• Initiation Set $\mathcal{I}_\omega \subseteq \mathcal{S}$: States where the option can be launched.
• Internal Policy $\pi_\omega(a \mid s)$: Maps states to primitive actions while the option is active.
• Termination Function $\beta_\omega(s) \in [0, 1]$: The probability that the option finishes in state $s$.

A high-level policy over options $\pi_{\Omega}(\omega \mid s)$ chooses an option, which executes for $k$ timesteps until $\beta_\omega(s) = 1$, converting the problem into a Semi-Markov Decision Process (SMDP).

2. Multi-Agent Centralized Training with Decentralized Execution (CTDE)

In multi-agent systems with $N$ agents, training independent Q-learners fails because from Agent 1's perspective, the environment is non-stationary as Agent 2 updates its policy.

Under CTDE (e.g., QMIX, MAPPO):

• During Training: A centralized Critic has access to the full joint state $\mathbf{S} = (s_1, \dots, s_N)$ and all joint actions $\mathbf{a} = (a_1, \dots, a_N)$, computing a centralized total value $Q_{\text{tot}}(\mathbf{S}, \mathbf{a})$.
• During Execution: Individual agents execute decentralized policies $\pi_i(a_i \mid o_i)$ conditioned strictly on their local observation $o_i$, requiring zero runtime inter-agent communication.

3. Intrinsic Curiosity Module (ICM) (Pathak et al., 2017)

In reward-sparse environments, the agent generates its own intrinsic reward $r_t^i$ driven by prediction error over environment dynamics.

An embedding network maps states to feature representations: $\phi(S_t)$. A forward dynamics network predicts next-state features given current features and executed action: $\hat{\phi}(S_{t+1}) = f(\phi(S_t), A_t)$. The intrinsic reward bonus is the mean squared prediction error:

The agent's total reward becomes:

Unfamiliar states generate high prediction error, incentivizing the agent to explore novel dynamics even when extrinsic reward is zero.

4. Meta-Reinforcement Learning ($\text{RL}^2$, MAML)

Instead of training on a single MDP, the agent trains across a distribution of tasks $p(\mathcal{T})$. By encoding past transitions $(s_{t-1}, a_{t-1}, r_t)$ into the recurrent hidden state of an LSTM or Transformer policy, the network "learns to learn", adapting to new transition dynamics within a single rollout episode without further weight updates.

Worked Example

An agent explores a labyrinth with zero extrinsic reward ($R^{\text{extrinsic}} = 0.0$). The agent utilizes an Intrinsic Curiosity Module with scaling hyperparameter $\eta = 0.5$. Discount factor $\gamma = 0.9$.

At step $t$, the agent takes action $A_t$ in state $S_t$.

1. Feature Extraction:
   • The true next state $S_{t+1}$ passes through feature encoder $\phi$: $$\phi(S_{t+1}) = [1.0, 2.0, 0.0]$$
2. Forward Dynamics Prediction:
   • The forward model predicts features based on $S_t$ and $A_t$: $$\hat{\phi}(S_{t+1}) = [1.2, 1.6, 0.5]$$
3. Prediction Error Calculation:
   • Error vector: $e = \hat{\phi} - \phi = [0.2, -0.4, 0.5]$
   • Squared L2 norm: $$\|e\|_2^2 = (0.2)^2 + (-0.4)^2 + (0.5)^2 = 0.04 + 0.16 + 0.25 = 0.45$$
4. Intrinsic Reward Bonus: $$r_t^i = \frac{\eta}{2} \|e\|_2^2 = \frac{0.5}{2} \times 0.45 = 0.25 \times 0.45 = 0.1125$$
5. Combined Bellman Target: With $R^{\text{extrinsic}} = 0.0$ and Critic estimate $V(S_{t+1}) = 0.80$: $$y_t = (R_t^{\text{ext}} + r_t^i) + \gamma V(S_{t+1}) = (0.0 + 0.1125) + 0.9(0.80) = 0.1125 + 0.72 = 0.8325$$

The curiosity bonus provided a non-zero training gradient that pulls the agent toward the unexplored corridor.

Code

1from typing import Tuple2import numpy as np3
4def compute_curiosity_reward(5    phi_next_true: np.ndarray,6    phi_next_pred: np.ndarray,7    r_extrinsic: float = 0.0,8    eta: float = 0.5,9) -> Tuple[float, float]:10    """Computes intrinsic curiosity bonus from forward model prediction error."""11    # Squared L2 error between predicted and true feature representations12    diff = phi_next_pred - phi_next_true13    squared_error = float(np.sum(diff ** 2))14    15    # Intrinsic curiosity reward16    r_intrinsic = (eta / 2.0) * squared_error17    r_total = r_extrinsic + r_intrinsic18    return r_intrinsic, r_total19
20# Features matching worked example21phi_true = np.array([1.0, 2.0, 0.0], dtype=np.float64)22phi_pred = np.array([1.2, 1.6, 0.5], dtype=np.float64)23
24r_int, r_tot = compute_curiosity_reward(25    phi_next_true=phi_true,26    phi_next_pred=phi_pred,27    r_extrinsic=0.0,28    eta=0.5,29)30
31print(f"Intrinsic Reward r_i: {r_int:.4f}")32# -> Intrinsic Reward r_i: 0.112533
34print(f"Total Combined Reward: {r_tot:.4f}")35# -> Total Combined Reward: 0.1125

Watch Out For

The Quick Version

• Hierarchical RL divides decision-making into multi-step options ($\langle \mathcal{I}_\omega, \pi_\omega, \beta_\omega \rangle$) managed by a meta-policy, solving sparse long-horizon credit assignment.
• Multi-Agent RL uses Centralized Training with Decentralized Execution (CTDE) to stabilize learning while keeping execution independent.
• Intrinsic Curiosity rewards agents for predicting forward dynamics errors ($\|\hat{\phi} - \phi\|^2$), driving exploration in zero-reward environments.

What to Read Next

• Offline Reinforcement Learning
• Value-Based Methods in RL
• Markov Decision Processes