Multi-Agent DDPG (MADDPG)

Why Does This Exist?

When standard single-agent reinforcement learning algorithms (such as DQN or DDPG) are directly deployed in multi-agent environments—an approach known as Independent Q-Learning (IQL) or Independent DDPG—they almost always fail to converge.

The core failure mode is environmental non-stationarity. In a multi-agent system, every agent updates its policy simultaneously based on its own rewards. From the individual perspective of Agent 1, the environment is not stationary: the state transition probability distribution:

$P\left(s_{t+1} \mid s_t, a_{1, t}\right)$

changes constantly as Agent 2 and Agent 3 adapt their behaviors. Because traditional Q-learning relies on the Markov assumption that transition dynamics are stationary, experience collected in the past becomes obsolete. The replay buffer turns into a source of stale, contradictory data, destabilizing temporal-difference updates.

The opposite extreme—training a single centralized controller that takes the union of all observations and outputs joint actions—suffers from combinatorial action space explosion ($\mathcal{A}^N$). More critically, a centralized policy requires instant global communication across all agents during live execution, which is infeasible in decentralized robotics, autonomous driving, or competitive games.

Multi-Agent Deep Deterministic Policy Gradient (MADDPG), introduced by Ryan Lowe et al. in 2017, resolves this dilemma through Centralized Training with Decentralized Execution (CTDE) for continuous action spaces. During training in a lab or simulator, critics have access to global states and all peer actions, restoring stationarity. During deployment, the centralized critics are discarded, and actors make decisions autonomously using only their local observations.

Think of It Like This

How It Actually Works

Centralized Training with Decentralized Execution (CTDE)

Consider a multi-agent game with $N$ agents. The environment is formalized as a partially observable stochastic game defined by:

• Global state space $\mathcal{S}$
• Local observation spaces $\mathcal{O}_1, \dots, \mathcal{O}_N$
• Continuous action spaces $\mathcal{A}_1, \dots, \mathcal{A}_N$

At each timestep, agent $i$ receives a local observation $o_i \in \mathcal{O}_i$ and executes continuous action $a_i \in \mathcal{A}_i$. The global state is represented by $x = (o_1, \dots, o_N)$ (or ground-truth simulator states), and the joint action is $a = (a_1, \dots, a_N)$.

MADDPG equips each agent $i$ with two distinct neural networks:

1. Decentralized Actor $\mu_{\theta_i}(o_i)$: Parameterized by $\theta_i$, maps agent $i$'s local observation $o_i$ directly to a deterministic continuous action $a_i$.
2. Centralized Critic $Q_{\phi_i}(x, a_1, \dots, a_N)$: Parameterized by $\phi_i$, evaluates the expected return of agent $i$ conditioned on the global state $x$ and the joint action of all agents.

1EXECUTION (Deployment):2   o₁ ──> [ Actor μ_θ1 ] ──> a₁ ──┐3                                  ├──> Environment ──> r₁, r₂, o'₁, o'₂4   o₂ ──> [ Actor μ_θ2 ] ──> a₂ ──┘5
6TRAINING (Replay Buffer):7   Global State x, Joint Actions (a₁, a₂) ──> [ Centralized Critic Q_φ1 ]8                                                         │9                                               ∇_a1 Q_φ1 │ Policy Gradient10                                                         ▼11                                                  [ Actor μ_θ1 ]

Centralized Critic Optimization

Because the critic conditions on the joint action vector $a = (a_1, \dots, a_N)$, the environment transition probability:

$P\left(x' \mid x, a_1, \dots, a_N\right)$

is completely stationary, even while individual policies $\mu_{\theta_j}$ evolve.

Critic $i$ minimizes the mean squared Bellman error:

$\mathcal{L}(\phi_i) = \mathbb{E}_{x, a, r, x'} \left[ \left( Q_{\phi_i}(x, a_1, \dots, a_N) - y_i \right)^2 \right]$

where the target return $y_i$ is computed using target actor networks $\mu'_{\theta_j}$ and target critic network $Q'_{\phi_i}$:

$y_i = r_i + \gamma Q'_{\phi_i}\left(x',\, a'_1, \dots, a'_N\right) \Big|_{a'_j = \mu'_{\theta_j}(o'_j)}$

Each agent has its own individual reward function $r_i$, meaning MADDPG naturally supports cooperative games ($r_1 = r_2$), competitive zero-sum games ($r_1 = -r_2$), and mixed-motive environments.

Decentralized Deterministic Policy Gradient

To update the decentralized actor $\mu_{\theta_i}$, the gradient of the expected return $J(\mu_i) = \mathbb{E}[R_i]$ is calculated via the chain rule through the centralized critic:

$\nabla_{\theta_i} J(\mu_i) = \mathbb{E}_{x, a \sim \mathcal{D}} \left[ \nabla_{\theta_i} \mu_{\theta_i}(o_i) \cdot \nabla_{a_i} Q_{\phi_i}(x, a_1, \dots, a_N) \Big|_{a_i = \mu_{\theta_i}(o_i)} \right]$

Crucial Insight: The gradient $\nabla_{a_i} Q_{\phi_i}$ is taken strictly with respect to agent $i$'s own action $a_i$. Peer actions $a_{j \ne i}$ are passed into the critic as fixed environmental context. The actor updates its weights $\theta_i$ to push its continuous action $a_i$ in the direction that maximizes $Q_{\phi_i}$ given the current behaviors of all other agents.

Modeling Peer Policies and Policy Ensembles

Inferring Unknown Peer Actions

In competitive or uncooperative settings where peer actions are not broadcast during training, agent $i$ can train internal generative models $\hat{\mu}_j^i(o_j)$ to approximate peer policies by maximizing log-likelihood over historical transitions:

$\mathcal{L}(\hat{\mu}_j^i) = -\mathbb{E} \left[ \log \hat{\mu}_j^i(a_j \mid o_j) \right]$

The inferred action $\hat{a}_j = \hat{\mu}_j^i(o_j)$ replaces $a_j$ in the centralized critic evaluation.

Policy Ensembles for Adversarial Robustness

In competitive multi-agent games, an agent can easily overfit to the idiosyncratic weaknesses of a specific training partner. MADDPG counters this by training an ensemble of $K$ distinct sub-policies for each agent. In each training episode, a sub-policy is randomly sampled for each agent, forcing policies to develop generalizable, robust counter-strategies.

Worked numerical example

Let us trace a concrete update step for a 2-agent continuous control environment where Agent 1 learns to coordinate with Agent 2.

Environment Setup

• Global state: $x = [1.0, 2.0]$
• Local observations: $o_1 = 1.0$, $o_2 = 2.0$
• Actions executed: $a_1 = 0.50$, $a_2 = -0.50$
• Rewards received: $r_1 = 2.0$, $r_2 = 1.0$
• Next global state: $x' = [1.2, 1.8]$
• Next local observations: $o'_1 = 1.2$, $o'_2 = 1.8$
• Discount factor: $\gamma = 0.90$

Step 1: Compute Target Actions

The target actor networks evaluate the next local observations:

• $a'_1 = \mu'_{\theta_1}(o'_1) = 0.60$
• $a'_2 = \mu'_{\theta_2}(o'_2) = -0.40$

Step 2: Compute Centralized Critic Target ($y_1$)

The centralized target critic for Agent 1 evaluates the next state and joint target actions: $Q'_{\phi_1}(x', a'_1, a'_2) = Q'_{\phi_1}([1.2, 1.8],\, 0.60,\, -0.40) = 4.00$

Compute the Bellman target: $y_1 = r_1 + \gamma \cdot Q'_{\phi_1}(x', a'_1, a'_2) = 2.0 + 0.90 \times 4.00 = 2.0 + 3.60 = 5.60$

Step 3: Compute Critic TD Error and Loss

Current centralized critic output for the executed transition: $Q_{\phi_1}(x, a_1, a_2) = Q_{\phi_1}([1.0, 2.0],\, 0.50,\, -0.50) = 5.00$

The temporal difference (TD) error is: $\delta_1 = y_1 - Q_{\phi_1}(x, a_1, a_2) = 5.60 - 5.00 = +0.60$

The squared error loss for Critic 1 is: $\mathcal{L}(\phi_1) = \delta_1^2 = (0.60)^2 = 0.3600$

Step 4: Compute Decentralized Actor Gradient

Suppose backpropagation through Critic 1 reveals that increasing $a_1$ increases predicted joint return: $\nabla_{a_1} Q_{\phi_1}(x, a_1, a_2) = 1.50$

Meanwhile, the local sensitivity of Actor 1 with respect to its weights $\theta_1$ is: $\nabla_{\theta_1} \mu_{\theta_1}(o_1) = 0.80$

Applying the chain rule, the deterministic policy gradient for Actor 1 is: $\nabla_{\theta_1} J(\mu_1) = \nabla_{\theta_1} \mu_{\theta_1}(o_1) \cdot \nabla_{a_1} Q_{\phi_1} = 0.80 \times 1.50 = 1.20$

Actor 1 updates its parameters: $\theta_1 \leftarrow \theta_1 + \alpha \times 1.20$. Notice that Actor 1's gradient did not require calculating $\nabla_{a_2} Q_{\phi_1}$; it optimized purely its own action input.

Code

Below is a self-contained, type-hinted Python implementation demonstrating MADDPG's centralized critic Bellman target evaluation, temporal difference updates, deterministic actor gradients, and decentralized execution invariance:

1from dataclasses import dataclass2import math3from typing import Callable, List, Tuple4
5
6@dataclass7class MultiAgentTransition:8    """Stores experience tuples across N agents in continuous state-action space."""9
10    global_state: List[float]11    local_observations: List[float]12    joint_actions: List[float]13    rewards: List[float]14    next_global_state: List[float]15    next_local_observations: List[float]16
17
18class MADDPGStepEvaluator:19    """Evaluates MADDPG's Centralized Training with Decentralized Execution (CTDE) calculations."""20
21    def __init__(self, gamma: float = 0.90) -> None:22        self.gamma = gamma23
24    def compute_critic_td_target(25        self, reward_i: float, next_target_q_i: float26    ) -> float:27        """Computes Bellman target for centralized critic i: y_i = r_i + gamma * Q'_i(x', a'_1, ..., a'_N)."""28        return reward_i + self.gamma * next_target_q_i29
30    def compute_critic_td_error(31        self, current_q_i: float, target_y_i: float32    ) -> float:33        """Computes TD error: delta_i = y_i - Q_i(x, a_1, ..., a_N)."""34        return target_y_i - current_q_i35
36    def compute_actor_policy_gradient(37        self, grad_q_wrt_action_i: float, grad_actor_wrt_theta_i: float38    ) -> float:39        """Computes Deterministic Policy Gradient: grad_theta_i J = grad_theta_i mu_i(o_i) * grad_a_i Q_i(x, a)."""40        return grad_actor_wrt_theta_i * grad_q_wrt_action_i41
42    def verify_execution_invariance(43        self,44        actor_policy_fn: Callable[[float], float],45        local_obs_i: float,46        global_state_var_1: List[float],47        global_state_var_2: List[float],48    ) -> bool:49        """Verifies decentralized execution: actor action depends strictly on local observation o_i,50
51        independent of peer states or global telemetry.52        """53        action_env1 = actor_policy_fn(local_obs_i)54        action_env2 = actor_policy_fn(local_obs_i)55        return math.isclose(action_env1, action_env2)56
57
58if __name__ == "__main__":59    evaluator = MADDPGStepEvaluator(gamma=0.90)60
61    # 1. Worked Numerical Example: 2-Agent Continuous Control Step62    # Environment variables63    r_1 = 2.064    q_target_1 = 4.065    current_q_1 = 5.0066
67    # Critic Target & TD Error Calculation68    target_y_1 = evaluator.compute_critic_td_target(69        reward_i=r_1, next_target_q_i=q_target_170    )71    td_error_1 = evaluator.compute_critic_td_error(72        current_q_i=current_q_1, target_y_i=target_y_173    )74    critic_loss_1 = td_error_1**275
76    print("=== Centralized Critic 1 Update ===")77    print(f"Agent 1 Reward r_1:                {r_1:.2f}")78    print(f"Target Joint Q'_1(x', a'):         {q_target_1:.2f}")79    print(f"Target y_1 = r_1 + gamma * Q'_1:   {target_y_1:.2f}")80    print(f"Current Joint Q_1(x, a):           {current_q_1:.2f}")81    print(f"TD Error delta_1:                  {td_error_1:+.2f}")82    print(f"Critic 1 MSE Loss:                 {critic_loss_1:.4f}")83
84    assert math.isclose(target_y_1, 5.60)85    assert math.isclose(td_error_1, 0.60)86    assert math.isclose(critic_loss_1, 0.36)87
88    # 2. Decentralized Deterministic Policy Gradient89    grad_q_wrt_a1 = 1.5090    grad_mu1_wrt_theta = 0.8091    policy_grad_1 = evaluator.compute_actor_policy_gradient(92        grad_q_wrt_action_i=grad_q_wrt_a1,93        grad_actor_wrt_theta_i=grad_mu1_wrt_theta,94    )95
96    print("\n=== Decentralized Actor 1 Gradient ===")97    print(f"dQ_1 / da_1 (critic gradient):     {grad_q_wrt_a1:.2f}")98    print(f"dmu_1 / dtheta_1 (actor sensitivity): {grad_mu1_wrt_theta:.2f}")99    print(f"Policy Gradient grad_theta_1 J:    {policy_grad_1:.2f}")100
101    assert math.isclose(policy_grad_1, 1.20)102
103    # 3. Decentralized Execution Invariance Check104    # Actor 1 policy is a function purely of local observation: a_1 = 0.5 * o_1105    actor1_fn = lambda o1: 0.5 * o1106    is_invariant = evaluator.verify_execution_invariance(107        actor_policy_fn=actor1_fn,108        local_obs_i=1.0,109        global_state_var_1=[1.0, 2.0],110        global_state_var_2=[1.0, 99.0],111    )112
113    print("\n=== Decentralized Execution Verification ===")114    print(f"Actor Execution Invariant to Peer States: {is_invariant}")115    assert is_invariant116
117    print("\nAll MADDPG step calculations verified successfully!")

Expected output:

1=== Centralized Critic 1 Update ===2Agent 1 Reward r_1:                2.003Target Joint Q'_1(x', a'):         4.004Target y_1 = r_1 + gamma * Q'_1:   5.605Current Joint Q_1(x, a):           5.006TD Error delta_1:                  +0.607Critic 1 MSE Loss:                 0.36008
9=== Decentralized Actor 1 Gradient ===10dQ_1 / da_1 (critic gradient):     1.5011dmu_1 / dtheta_1 (actor sensitivity): 0.8012Policy Gradient grad_theta_1 J:    1.2013
14=== Decentralized Execution Verification ===15Actor Execution Invariant to Peer States: True16
17All MADDPG step calculations verified successfully!

Watch Out For

The Quick Version

• Centralized Training with Decentralized Execution (CTDE): MADDPG trains centralized critics with access to global state and all peer actions, but deploys actors that operate purely on local partial observations.
• Resolving Non-Stationarity: Conditioning the critic on joint actions $a = (a_1, \dots, a_N)$ restores environment stationarity during training, stabilizing multi-agent temporal-difference learning.
• Independent Policy Gradients: During actor optimization, policy gradients backpropagate strictly through agent $i$'s own action ($\nabla_{a_i} Q_{\phi_i}$), treating peer actions as fixed environmental context.
• General Multi-Agent Motives: Unlike purely cooperative algorithms that enforce shared team rewards, MADDPG supports arbitrary reward functions, excelling across cooperative, competitive, and mixed-motive continuous environments.

What to Read Next

• Multi-Agent RL
• Multi-Agent PPO (MAPPO)
• Value Decomposition Networks (VDN)