Core ML

Proximal Policy Optimization

Reuse the same batch of experience for several updates, but clip the objective so no single update can push the policy too far from where it started.

The clipped objective tracks the raw policy-ratio objective closely near a ratio of one, then flattens outside a narrow band, capping how much a single update can push the policy away from where it started

Why Does This Exist?

Policy gradient methods already names the core problem: gradient estimates from a policy are noisy, and each batch of collected experience only describes the current policy — one update changes the policy, which means the data you just collected is now slightly stale. That leaves a narrow path to walk: take a large step, and the noisy gradient estimate can push the policy somewhere much worse, from which it may never recover; take a tiny step, and training crawls, since fresh experience has to be collected before the next tiny step can happen at all.

Proximal Policy Optimization directly targets that path. It reuses a batch of collected experience for several update steps — getting more value out of every batch than a single gradient step would — while capping how far any single update is allowed to move the policy, so an unlucky noisy batch can't cause lasting damage.

Think of It Like This

Revising an essay draft, but never past a certain amount per pass

An editor reviewing a draft could rewrite entire paragraphs in one pass based on their first read — fast, but if their read was slightly off, the rewrite might make things worse in ways that are hard to undo. Instead, the editor works in bounded passes: mark up the sections that seem weakest, revise them by a limited amount, then read the whole draft again fresh before deciding what to revise next.

Each pass makes real progress, and no single pass can swing the draft wildly based on one reviewer's momentary misjudgment. Proximal Policy Optimization applies the same discipline to updating a policy: real progress each step, capped so one noisy batch can't do lasting harm.

How It Actually Works

The probability ratio measures how far the policy has moved

For an action $a$ taken in state $s$ under the policy in effect when the data was collected (the "old" policy $\pi_{\theta_{\text{old}}}$ ), define the ratio between the new and old policy's probability of that same action:

r(\theta) = \frac{\pi_\theta(a \mid s)}{\pi_{\theta_{\text{old}}}(a \mid s)}

$r(\theta) = 1$ means the new policy agrees exactly with the old one on this action. $r(\theta)$ far from 1 means the policy has already moved a long way from where the data was collected — exactly the situation that makes the collected data unreliable for estimating the current policy's gradient.

The clipped objective removes the incentive to overshoot

The raw policy-gradient objective, scaled by this ratio, would keep rewarding a larger and larger ratio for a good action indefinitely — nothing in it says "far enough." PPO's clipped objective caps that reward:

L(\theta) = \mathbb{E}\Big[\min\big(r(\theta) A,\ \text{clip}(r(\theta), 1-\epsilon, 1+\epsilon)\, A\big)\Big]

$A$ is the advantage — how much better this action turned out than expected, as covered in policy gradient methods — and $\epsilon$ is typically 0.1 to 0.2. Taking the minimum of the raw and clipped terms means that once the ratio strays outside $[1-\epsilon, 1+\epsilon]$ , the objective stops offering any further reward for moving the policy even further in that direction. The update still happens, but the incentive to overshoot disappears exactly at the boundary.

Why this makes reusing a batch safe

Because moving too far from the old policy no longer earns extra objective value, running several gradient steps on the same batch of collected experience doesn't run away — each additional step naturally settles rather than compounding a runaway ratio. This is what lets PPO extract several epochs of learning from a single batch of environment interaction, instead of collecting fresh data after every single update the way a strict on-policy method otherwise would need to.

Why PPO is the workhorse behind RLHF

Reinforcement learning from human feedback trains a language model's policy against a learned reward model, and every reward evaluation and every rollout is expensive — generating a full completion, scoring it, and doing this at scale. Reusing a batch of generated rollouts for multiple update steps, safely, is exactly what makes RLHF's compute budget survivable, which is the specific reason PPO rather than plain REINFORCE became the default optimizer in that setting.

Show Me the Code

The clipped objective compared against the raw ratio-scaled objective, across a range of ratios.

1import numpy as np2
3advantage = 1.04eps = 0.25ratios = np.array([0.5, 0.85, 1.0, 1.15, 1.5, 2.5])6unclipped = ratios * advantage7clipped = np.clip(ratios, 1 - eps, 1 + eps) * advantage8objective = np.minimum(unclipped, clipped)9for r, o in zip(ratios, objective):10    print(f"ratio={r:.2f}  objective={o:.2f}")11# -> ratio=0.50  objective=0.5012# -> ratio=0.85  objective=0.8513# -> ratio=1.00  objective=1.0014# -> ratio=1.15  objective=1.1515# -> ratio=1.50  objective=1.2016# -> ratio=2.50  objective=1.20

Inside the band from 0.8 to 1.2, the objective tracks the ratio exactly. Past 1.2, the objective flattens at 1.20 regardless of how much further the ratio climbs — a ratio of 2.5 earns no more reward than a ratio of 1.5, which removes any incentive to keep pushing the policy that far.

Watch Out For

Assuming a larger epsilon always trains faster

A wider clipping band lets each update move the policy further per step, which sounds like faster progress. In practice, a band too wide reintroduces the instability PPO exists to prevent — an unlucky batch can now push the policy substantially before the next batch corrects course. The commonly used range of 0.1 to 0.2 is a reasonable default, and a wider setting deserves a specific reason, not a default assumption that wider is faster.

Running too many epochs over one batch

The clipped objective makes reusing a batch several times safe, not infinitely reusable. Running enough epochs over the same batch will eventually still overfit the policy to that specific batch's noise, and the ratio-clipping only bounds how far a single update can move — it doesn't prevent many small moves from adding up over many epochs. Three to ten epochs per batch is the typical range; treat epoch count as a hyperparameter to tune, not a fixed constant.

The Quick Version

The probability ratio compares the new policy's likelihood of a past action to the old policy's likelihood of that same action, measuring how far the policy has already moved.
The clipped objective stops rewarding further movement once the ratio leaves a narrow band around 1, removing the incentive to overshoot.
This makes it safe to run several gradient update steps on the same batch of collected experience, rather than collecting fresh data after every single step.
PPO's ability to reuse expensive rollouts safely is why it became the default optimizer for reinforcement learning from human feedback.