Actor Critic Methods

Actor-critic methods grade each action the moment it lands instead of waiting for the episode score. A learned critic supplies the grade, which cuts the noise that makes plain policy gradients so slow.

A one-step TD grade replaces the noisy full-episode return, so the actor learns from a delta of 1.49

Why Does This Exist?

REINFORCE is the simplest policy gradient: run an episode, add up the return, push every action toward or away from what happened. It works, and it is honest. It is also ruinously noisy, because the grade for your first move contains every lucky bounce from the rest of the episode. Two identical opening moves can get wildly different grades. Training needs mountains of episodes to average that noise away.

Value methods dodge the noise by bootstrapping off the next state, but a Q-learner cannot output a continuous torque or a genuinely stochastic policy. So continuous control and noisy gradients looked like a forced choice.

Actor-critic methods refuse the choice. A critic learns $V(s)$ , and the TD error $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$ replaces the full-episode return as the actor grade. The grade still points uphill on average, but its noise comes from one transition instead of hundreds. That single swap is what lets one family cover discrete games, robot joints, A2C, PPO, and SAC.

This page teaches the update rules and why the variance drops. The wiring that carries them lives on the actor-critic architecture page.

Think of It Like This

Learning darts with a friend calling each throw

You throw darts at a board. Under REINFORCE rules, you throw all twenty darts, someone totals the score, and one number judges every throw including the good ones buried in a bad round. You learn almost nothing per visit.

Now a friend stands beside the board and calls each throw as it lands: that one grouped tight, this one drifted low. Each throw gets its own grade while your arm still remembers the motion. You adjust the next throw immediately instead of reconstructing the evening from a total.

Where it stops: your friend starts out guessing too. Early grades reflect their inexperience as much as your aim, and if you only ever throw the way they praise, you both settle into a comfortable mediocrity. That is exactly the critic-bias and premature-convergence failure below.

How It Actually Works

The actor is a policy $\pi_\theta(a \mid s)$ with weights $\theta$ . The critic is a value estimate $V_w(s)$ with weights $w$ . Each transition $(s_t, a_t, r_t, s_{t+1})$ runs three computations. Here $\gamma$ discounts the future and both updates share the same error:

The grade

\delta_t = r_t + \gamma V_w(s_{t+1}) - V_w(s_t)

Read it as surprise: reward plus what the next state is worth, minus what you expected. Positive means the action beat the forecast.

The two updates

Critic. Shrink the squared surprise, $\tfrac{1}{2}\delta_t^2$ , so forecasts improve.
Actor. Step along $\delta_t \, \nabla_\theta \log \pi_\theta(a_t \mid s_t)$ , which raises the taken action probability exactly when the surprise is positive.

Why the noise drops

REINFORCE grades with the full return $G_t$ , whose randomness spans the whole rest of the episode. The TD grade swaps $G_t$ for $r_t + \gamma V(s_{t+1})$ : one real reward plus a learned forecast. The forecast carries no sampling noise at all, so the only randomness left is a single transition. Bias creeps in because the forecast can be wrong, and that bias-for-variance trade is the defining deal of the family. On-policy members (A2C, PPO) keep the deal honest by grading fresh behavior; off-policy members (DDPG, SAC) reuse old transitions and pay for it with correction machinery.

Worked example

Reward $r = 2.0$ , current estimate $V(s) = 1.5$ , next estimate $V(s') = 1.0$ , discount $\gamma = 0.99$ , step size $\alpha = 0.05$ :

\delta = 2.0 + 0.99 \times 1.0 - 1.5 = 2.0 + 0.99 - 1.5 = 1.49

The action beat the forecast by 1.49. The critic target is $2.99$ and the estimate moves:

V(s) \leftarrow 1.5 + 0.05 \times 1.49 = 1.5745

The actor raises the taken action log-probability in proportion to $0.05 \times 1.49 = 0.0745$ . A full-episode return would have mixed this reward with everything after it; the TD grade isolates this step contribution.

Code

One transition through both updates, in standard-library Python, replaying the worked numbers:

1def actor_critic_step(2    v: float, v_next: float, reward: float, gamma: float = 0.99, alpha: float = 0.053) -> tuple:4    """Return (td error, bootstrapped target, updated value) for one transition."""5    target = reward + gamma * v_next6    delta = target - v7    return delta, target, v + alpha * delta8
9
10delta, target, v_new = actor_critic_step(1.5, 1.0, 2.0)11print(f"{delta:.2f}")12# -> 1.4913print(f"{target:.2f}")14# -> 2.9915print(f"{v_new:.4f}")16# -> 1.5745

Each print matches the hand arithmetic above.

Watch Out For

The critic gradients trampling the actor features

When both halves share early layers, the critic mean-squared-error gradient is usually larger than the actor gradient and rewrites the shared features to serve value prediction, wrecking a decent policy overnight. The symptom is value loss falling while episode return collapses. The fix is separate networks, or scaling the critic loss down and stopping actor gradients from entering shared layers. If your actor dies just as the critic gets good, this interference is the first suspect.

A critic that flatters, and an actor that believes it

Value estimates drift upward because the policy chases whatever the critic rates highest and the critic then trains on those same chosen actions. The symptom is predicted values climbing while real returns stall or sink. Soft Actor-Critic and TD3 run two critics and train against the smaller prediction, which cuts the flattery loop. Whenever predicted and achieved values diverge, distrust the critic first.

The Quick Version

The TD error $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$ replaces the full-episode return as the actor grade.
One transition of noise beats hundreds of transitions of noise, at the price of some critic bias.
The critic minimizes $\tfrac{1}{2}\delta_t^2$ while the actor climbs $\delta_t \nabla_\theta \log \pi_\theta$ .
The worked step gives $\delta = 1.49$ and moves $V(s)$ from 1.5 to 1.5745.
A2C and PPO stay on-policy to keep grades honest; SAC and DDPG reuse old data with extra correction machinery.
Modern actor-critic variants differ mostly in how they tame critic bias and reuse data.