Policy Gradient Methods

Why Does This Exist?

A robotic arm on a parcel sorting line picks up boxes arriving at a slightly different angle every time. Its actions are real numbers: wrist angle in degrees, gripper force in newtons, descent speed. No list of options to score.

Q-learning needs that list, because taking a maximum means enumerating. Ten buckets of wrist angle throw away the precision that makes a grasp hold. A thousand buckets times two more dimensions, and the count passes a billion.

Second reason: sometimes the best behaviour is genuinely random. With a parcel's pose only roughly known, a controller that always tries the identical grip either always succeeds on that box shape or always fails.

So stop learning what actions are worth. Optimise the policy itself.

Think of It Like This

How It Actually Works

Score the action, then push its probability

The policy is a network: state in, parameters of a distribution over actions out. For continuous control that is usually a mean and a spread per dimension, so wrist angle is drawn from a bell centred wherever the network thinks it belongs. Exploration comes free, and the policy can narrow that spread itself.

You maximise expected return $J(\theta)$, with $\theta$ the network's weights. The policy gradient theorem says the gradient of that return is an average over behaviour the policy actually produced:

$\pi_{\theta}(a \mid s)$ is the density the policy puts on action $a$ in state $s$, $\nabla_{\theta} \log \pi_{\theta}$ is the direction in weight space that makes it more likely, and $G$ is the discounted return that followed. REINFORCE is that expression, sampled.

Notice what is absent: nothing differentiates through the conveyor. You need only your own network's gradient, which is why this works on black boxes and real hardware.

The wall: an unbiased estimate you cannot use

It is unbiased, and it converges given enough episodes. The trouble is how many enough is.

One sorting cycle is forty actions and one return, and all forty get multiplied by the same number. A textbook grasp in a cycle where the conveyor jammed downstream gets a negative weight and is trained away; a sloppy grasp on a lucky cycle gets reinforced.

Scale makes it worse. Say a cycle earns between 190 and 210. Every action gets multiplied by roughly 200, a constant that says nothing about any individual action while inflating every sample's variance. Monte Carlo estimation has a fixed rate: halve the noise, quadruple the samples. Here, samples are grasps.

Baselines, critics, and a leash on the update

Everything after REINFORCE is variance reduction wearing a different hat.

A baseline subtracts $b(s)$, anything that depends on the state but not the action. The expectation does not move: the expected value of $\nabla_{\theta} \log \pi_{\theta}$ under the policy's own actions is zero, so an action-independent factor vanishes in the average. Variance falls hard.

The natural $b(s)$ is the state value $V(s)$, what you would normally get from here. Then $G - V(s)$ is the advantage: how much better this action turned out than usual. Absolute number becomes comparison, the same move the kitchen made. Actor-critic learns $V(s)$ with a second network while the policy trains, as actor-critic methods describe.

A large update also changes the data you collect next, so an over-eager step can leave the policy gathering useless experience. PPO limits the move bluntly: take the ratio of new to old probability for an action, and stop granting credit once it leaves a narrow band around one.

The bill is sample efficiency: that expectation is over the current policy, so one step leaves your data describing a policy that no longer exists. Q-learning replays a transition dozens of times; this spends a batch once. Sorting arms learn in simulation for a reason.

Code

The point is the check: a one-step policy whose return carries a large constant offset, with and without a mean baseline.

1import random2
3
4def grad_samples(baseline: float, trials: int = 20000) -> list[float]:5    """One-step Gaussian policy at mean 0.5; the return peaks at 2.0, plus a flat offset."""6    rng = random.Random(3)7    out: list[float] = []8    for _ in range(trials):9        a: float = rng.gauss(0.5, 1.0)10        ret: float = 40.0 - (a - 2.0) ** 2  # the offset is the point: it carries no information11        out.append((a - 0.5) * (ret - baseline))  # score gradient times return12    return out13
14
15def summarize(samples: list[float]) -> tuple[float, float]:16    mean: float = sum(samples) / len(samples)17    var: float = sum((x - mean) ** 2 for x in samples) / len(samples)18    return mean, var**0.519
20
21average_return: float = 40.0 - (0.5 - 2.0) ** 2 - 1.0  # what this policy earns on average22for b, name in ((0.0, "no baseline"), (average_return, "mean baseline")):23    m, s = summarize(grad_samples(b))24    print(f"{name:14s} mean {m:+.3f}   spread {s:.3f}")25    # -> no baseline    mean +2.806   spread 35.10926    # -> mean baseline  mean +3.046   spread 5.396

The true gradient is exactly 3.0 and both estimates sit near it: unbiased, as promised. The spread drops from 35.1 to 5.4, more than sixfold, purely from subtracting a number that says nothing about the action taken.

Watch Out For

The Quick Version

• A network outputs a distribution over actions; you do gradient ascent on expected return. Continuous actions need no discretisation.
• The gradient is the log-probability gradient of the action taken, weighted by the return that followed. REINFORCE is that, sampled.
• Nothing differentiates through the environment.
• Unbiased, and brutally noisy: one return is shared across every action in the episode, so good actions in bad episodes get punished.
• Our offset test reads means near the true 3.0 with spreads 35.1 without a baseline and 5.4 with one.
• A baseline cuts variance sharply. Set it to $V(s)$ for the advantage; learn it with a second network for actor-critic.
• Limit each update, because it changes the data you collect next. PPO clips the probability ratio.
• On-policy, so data goes stale every step: orders of magnitude more interaction than off-policy value learning.

What to Read Next

• Q-Learning and DQN
• Actor Critic Methods
• Policy Gradient Theorem