Skip to content
AI360XpertBeta

Proximal Policy Optimization

Reuse one batch of rollouts for several updates, but clip the objective so no step can drag the policy far from the data it learned on.

With advantage 1.0 and band 0.8 to 1.2, ratios 1.5 and 2.5 both pay 1.2, so the objective flattens past the band
With advantage 1.0 and band 0.8 to 1.2, ratios 1.5 and 2.5 both pay 1.2, so the objective flattens past the band

Why Does This Exist?

Policy gradient methods name the bind. Gradient estimates from rollouts are noisy, and each batch describes only the policy that collected it. Take a large step on one noisy batch and the policy can land somewhere much worse, then collect its next batch from that worse place. Take tiny steps and training crawls, because every step waits on fresh environment interaction.

The setup underneath is reinforcement learning foundations: states, actions, rewards, and a discount that trades now against later. PPO keeps that setup and changes the update: squeeze several gradient steps out of each batch, while capping how far any step may move the policy. More learning per rollout, less damage per unlucky batch.

Think of It Like This

Edit passes with a leash

An editor gets one read of a draft, then makes revision passes without rereading. The first pass fixes clear typos. The second still helps. By the fifth pass on the same read, edits start fixing noise in that one reading rather than the draft.

So the editor adds a leash: each pass may move any paragraph only a little from the version that was actually read. Progress per read rises because several passes run per read, and no single stale read can rewrite the book. Fresh reads arrive before the leash binds.

The read is the rollout batch, each pass is a gradient epoch on that batch, the leash is the clip band around ratio 1, and rereading is collecting the next batch. PPO is the rule that enforces the leash.

Where it stops: an editor can reread cheaply, while a robot or a language model rollout costs time and money. The whole reason to reuse a batch is that fresh reads are dear, so the leash earns its keep only where data costs more than compute.

How It Actually Works

The ratio measures movement from the data

For action aa in state ss, write πθ\pi_{\theta} for the current policy and πθold\pi_{\theta_{old}} for the policy that collected the batch. The ratio is:

r(θ)=πθ(a∣s)πθold(a∣s)r(\theta) = \frac{\pi_{\theta}(a \mid s)}{\pi_{\theta_{old}}(a \mid s)}

Here πθ(a∣s)\pi_{\theta}(a \mid s) is the current likelihood of the past action, and the denominator is its likelihood under the old policy. A ratio of 1 means no movement. Far from 1 means the data no longer describes the policy being scored.

The clip removes the reward for overshoot

Write AA for the advantage, meaning how much better the action turned out than the state average, and ϵ\epsilon for the clip width, usually 0.1 to 0.2. The clipped objective is:

L(θ)=E[min⁡(r(θ)A, clip(r(θ),1−ϵ,1+ϵ)A)]L(\theta) = \mathbb{E}\left[\min\left(r(\theta) A, \, \mathrm{clip}(r(\theta), 1 - \epsilon, 1 + \epsilon) A\right)\right]

Here clip\mathrm{clip} pins the ratio into [1−ϵ,1+ϵ][1 - \epsilon, 1 + \epsilon], and E\mathbb{E} averages over the batch. Once the ratio leaves the band, the clipped term stops growing, and the minimum stops rewarding further movement.

Worked example: advantage 1.0, band 0.8 to 1.2

Set A=1.0A = 1.0 and ϵ=0.2\epsilon = 0.2, so the band is [0.8,1.2][0.8, 1.2]. The raw term is r×1.0r \times 1.0 and the clipped term pins rr first:

  • r=0.50r = 0.50 gives raw 0.50 and clipped 0.80, minimum 0.50.
  • r=0.85r = 0.85 gives 0.85 and 0.85, minimum 0.85.
  • r=1.00r = 1.00 gives 1.00 and 1.00, minimum 1.00.
  • r=1.15r = 1.15 gives 1.15 and 1.15, minimum 1.15.
  • r=1.50r = 1.50 gives raw 1.50 and clipped 1.20, minimum 1.20.
  • r=2.50r = 2.50 gives raw 2.50 and clipped 1.20, minimum 1.20.

Inside the band the objective tracks the ratio exactly. Past 1.2 it flattens at 1.20, so 2.5 earns nothing over 1.5. That flat roof is the diagram: a diagonal that turns horizontal at the band edge. For a bad action with A=−1.0A = -1.0, the same arithmetic gives r=0.5r = 0.5 paying −0.80-0.80 and r=1.5r = 1.5 paying −1.50-1.50, the pessimistic side of the same minimum. Because the roof caps the incentive, three to ten epochs on one batch settle instead of running away, which is why costly rollouts in contextual bandits and human feedback training survive reuse.

Code

The point is the check: rebuild the six ratios from the worked example and confirm the flat roof at 1.20.

def clipped_objective(ratio: float, advantage: float, eps: float = 0.2) -> float:    """PPO surrogate for one ratio: raw against clipped, keep the smaller."""    clipped: float = min(1 + eps, max(1 - eps, ratio))    return min(ratio * advantage, clipped * advantage)

ratios: list[float] = [0.5, 0.85, 1.0, 1.15, 1.5, 2.5]for r in ratios:    print(f"ratio={r:.2f} objective={clipped_objective(r, 1.0):.2f}")    # -> ratio=0.50 objective=0.50    # -> ratio=0.85 objective=0.85    # -> ratio=1.00 objective=1.00    # -> ratio=1.15 objective=1.15    # -> ratio=1.50 objective=1.20    # -> ratio=2.50 objective=1.20

The printout matches the hand table: exact tracking to 1.15, then 1.20 twice. Change ϵ\epsilon to 0.1 and watch the roof drop to 1.10, which shows what the width controls.

Watch Out For

A wide clip that reintroduces the crash

Training runs faster for an hour, then one batch sends the policy off a cliff and it never recovers.

Symptom: return climbs, then drops vertically after a single update, with ratios far past 2.0 in the logs. Fix: keep ϵ\epsilon near 0.1 to 0.2 unless a sweep says otherwise, and treat band width as stability control rather than speed control. Wider is not faster once a crash costs a day.

Too many epochs on one stale batch

Loss keeps falling on the same batch while live return stalls, then falls.

Symptom: batch loss improves for twenty epochs while the next fresh batch scores worse, a classic overfit gap. Fix: hold epochs near three to ten, refresh the batch on schedule, and tune epoch count like a learning rate. Clipping bounds each step, not the sum of many small steps.

The Quick Version

  • The ratio r(θ)r(\theta) divides current action likelihood by the old likelihood, measuring movement from the collecting policy.
  • The clipped objective keeps the smaller of the raw and band pinned terms, removing the reward for leaving [0.8,1.2][0.8, 1.2] when ϵ=0.2\epsilon = 0.2.
  • Our six ratios at advantage 1.0 pay 0.50, 0.85, 1.00, 1.15, 1.20, and 1.20, so 2.5 earns nothing over 1.5.
  • The flat roof lets one batch serve several epochs without running away.
  • Reusing dear rollouts safely is why PPO carries human feedback training.
  • Band width and epoch count stay small: clipping caps steps, not their sum.