Policy Gradient Methods

Policy gradients tune the policy itself by climbing expected return, so they handle continuous and stochastic actions where picking a max Q value fails.

A policy gradient update lifts the sampled action from probability 0.50 to 0.73 by stepping along return times score function

Why Does This Exist?

Picking $\arg\max_a Q(s, a)$ breaks when actions are continuous. A robot arm chooses joint torques from an infinite set, so there is no list to max over. Policy gradients dodge the problem: they parameterize the policy $\pi_\theta(a \mid s)$ directly and climb expected return $J(\theta)$ with respect to $\theta$ . The symbol $\theta$ holds the policy weights, and $J(\theta)$ is the expected return those weights earn.

There is a second motive. Some tasks demand randomness, like mixing strategies so an opponent cannot exploit you. Value methods produce greedy deterministic policies, while a parameterized policy represents randomness naturally. This page covers the gradient itself; REINFORCE turns it into an algorithm, actor critic methods tame its variance, and policy objective functions define exactly what $J$ means.

Think of It Like This

A food truck owner testing two specials

A food truck owner sells a noodle bowl and a taco plate. Each morning she picks the split of ingredients by gut feel, then counts profit at close. A great noodle day means: shift tomorrow's split toward noodles. A bad taco day means: shift away from tacos.

She never writes down the value of every possible split. She just nudges the current split toward whatever sold well, in proportion to how well it sold. Returns play the role of profit, and the score function plays the role of the nudge direction. Over weeks the split settles where profit peaks.

The analogy stops at credit assignment: her daily profit mixes both dishes, while the real update separates each action's contribution through its own log probability gradient.

How It Actually Works

The goal is $J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[G(\tau)]$ , the expected return over trajectories the current policy generates. The policy gradient theorem states the gradient without differentiating the environment at all:

\nabla_\theta J(\theta) = \mathbb{E}\left[ \sum_t \nabla_\theta \log \pi_\theta(a_t \mid s_t) \, G_t \right]

In words: for each visited state, take the direction that raises the taken action's log probability, and scale it by the return $G_t$ that followed. Good outcomes pull the policy toward the actions that preceded them; bad ones push away. The term $\nabla_\theta \log \pi_\theta(a_t \mid s_t)$ is called the score function, and $G_t$ is the discounted return from step $t$ .

Raw returns are noisy, so practitioners subtract a baseline $b(s_t)$ , often a learned state value, forming the advantage $A_t = G_t - b(s_t)$ . The baseline leaves the expected gradient unchanged but shrinks its variance. And because the update samples actions from the policy, it works unchanged for continuous action spaces: a Gaussian policy just outputs a mean and spread instead of softmax logits.

Worked example

Two lunch specials, one parameter pair $\theta = [0.0, 0.0]$ . Softmax turns logits into probabilities, so $\pi = [0.5, 0.5]$ . The truck pushes noodles (action 1) and earns return $G = 10$ .

The score function for the taken action is one-hot minus probabilities: $[1 - 0.5, 0 - 0.5] = [0.5, -0.5]$ . Scale by the return: $10 \times [0.5, -0.5] = [5.0, -5.0]$ . Step with learning rate $\alpha = 0.1$ :

\theta \leftarrow [0.0, 0.0] + 0.1 \times [5.0, -5.0] = [0.5, -0.5]

New probabilities: $e^{0.5} \approx 1.6487$ and $e^{-0.5} \approx 0.6065$ , summing to $2.2552$ , so $\pi \approx [0.731, 0.269]$ . One good day moved noodles from $0.50$ to $0.73$ . That single stepped update is exactly what the diagram draws.

Code

This snippet replays the worked example with plain arithmetic, no libraries needed.

1import math2
3
4def softmax(logits):5    exps = [math.exp(z) for z in logits]6    total = sum(exps)7    return [e / total for e in exps]8
9
10theta = [0.0, 0.0]11probs = softmax(theta)12action, G, alpha = 0, 10.0, 0.113score = [(-p if i != action else 1.0 - p) for i, p in enumerate(probs)]14theta = [t + alpha * g * s for t, s in zip(theta, score)]15print([round(p, 2) for p in probs])16print([round(t, 1) for t in theta])17print([round(p, 2) for p in softmax(theta)])

Watch Out For

Trusting raw Monte Carlo returns

Returns swing wildly between episodes, so the gradient estimate swings with them. Training crawls or diverges even when the code is correct.

Symptom: updates alternate sign and the policy jitters instead of climbing. Fix: subtract a baseline, typically a fitted $V(s)$ , and update on advantages. When that still wobbles, move to actor critic updates that bootstrap instead of waiting for full returns.

Stepping so far the policy collapses

One large step can slam an action probability near 1, and sampling then never recovers the alternatives. The policy looks confident and performs badly.

Symptom: entropy crashes in a few updates while return falls. Fix: keep steps small, normalize advantages, and clip the update size the way trust region methods do. A policy you cannot sample from is a policy you cannot improve.

The Quick Version

Policy gradients climb $J(\theta)$ directly through $\nabla_\theta \log \pi_\theta$ scaled by return.
They handle continuous and stochastic actions where $\arg\max Q$ cannot even be written down.
The score function points toward the taken action; the return decides how hard to push.
Baselines turn returns into advantages, cutting variance without biasing the gradient.
Here one update with return 10 moved the sampled action from 0.50 to 0.73.