Proximal Policy Optimization
Reuse one batch of rollouts for several updates, but clip the objective so no step can drag the policy far from the data it learned on.
Why Does This Exist?
Policy gradient methods name the bind. Gradient estimates from rollouts are noisy, and each batch describes only the policy that collected it. Take a large step on one noisy batch and the policy can land somewhere much worse, then collect its next batch from that worse place. Take tiny steps and training crawls, because every step waits on fresh environment interaction.
The setup underneath is reinforcement learning foundations: states, actions, rewards, and a discount that trades now against later. PPO keeps that setup and changes the update: squeeze several gradient steps out of each batch, while capping how far any step may move the policy. More learning per rollout, less damage per unlucky batch.
Think of It Like This
Edit passes with a leash
An editor gets one read of a draft, then makes revision passes without rereading. The first pass fixes clear typos. The second still helps. By the fifth pass on the same read, edits start fixing noise in that one reading rather than the draft.
So the editor adds a leash: each pass may move any paragraph only a little from the version that was actually read. Progress per read rises because several passes run per read, and no single stale read can rewrite the book. Fresh reads arrive before the leash binds.
The read is the rollout batch, each pass is a gradient epoch on that batch, the leash is the clip band around ratio 1, and rereading is collecting the next batch. PPO is the rule that enforces the leash.
Where it stops: an editor can reread cheaply, while a robot or a language model rollout costs time and money. The whole reason to reuse a batch is that fresh reads are dear, so the leash earns its keep only where data costs more than compute.
How It Actually Works
The ratio measures movement from the data
For action in state , write for the current policy and for the policy that collected the batch. The ratio is:
Here is the current likelihood of the past action, and the denominator is its likelihood under the old policy. A ratio of 1 means no movement. Far from 1 means the data no longer describes the policy being scored.
The clip removes the reward for overshoot
Write for the advantage, meaning how much better the action turned out than the state average, and for the clip width, usually 0.1 to 0.2. The clipped objective is:
Here pins the ratio into , and averages over the batch. Once the ratio leaves the band, the clipped term stops growing, and the minimum stops rewarding further movement.
Worked example: advantage 1.0, band 0.8 to 1.2
Set and , so the band is . The raw term is and the clipped term pins first:
- gives raw 0.50 and clipped 0.80, minimum 0.50.
- gives 0.85 and 0.85, minimum 0.85.
- gives 1.00 and 1.00, minimum 1.00.
- gives 1.15 and 1.15, minimum 1.15.
- gives raw 1.50 and clipped 1.20, minimum 1.20.
- gives raw 2.50 and clipped 1.20, minimum 1.20.
Inside the band the objective tracks the ratio exactly. Past 1.2 it flattens at 1.20, so 2.5 earns nothing over 1.5. That flat roof is the diagram: a diagonal that turns horizontal at the band edge. For a bad action with , the same arithmetic gives paying and paying , the pessimistic side of the same minimum. Because the roof caps the incentive, three to ten epochs on one batch settle instead of running away, which is why costly rollouts in contextual bandits and human feedback training survive reuse.
Code
The point is the check: rebuild the six ratios from the worked example and confirm the flat roof at 1.20.
def clipped_objective(ratio: float, advantage: float, eps: float = 0.2) -> float: """PPO surrogate for one ratio: raw against clipped, keep the smaller.""" clipped: float = min(1 + eps, max(1 - eps, ratio)) return min(ratio * advantage, clipped * advantage)
ratios: list[float] = [0.5, 0.85, 1.0, 1.15, 1.5, 2.5]for r in ratios: print(f"ratio={r:.2f} objective={clipped_objective(r, 1.0):.2f}") # -> ratio=0.50 objective=0.50 # -> ratio=0.85 objective=0.85 # -> ratio=1.00 objective=1.00 # -> ratio=1.15 objective=1.15 # -> ratio=1.50 objective=1.20 # -> ratio=2.50 objective=1.20The printout matches the hand table: exact tracking to 1.15, then 1.20 twice. Change to 0.1 and watch the roof drop to 1.10, which shows what the width controls.
Watch Out For
A wide clip that reintroduces the crash
Training runs faster for an hour, then one batch sends the policy off a cliff and it never recovers.
Symptom: return climbs, then drops vertically after a single update, with ratios far past 2.0 in the logs. Fix: keep near 0.1 to 0.2 unless a sweep says otherwise, and treat band width as stability control rather than speed control. Wider is not faster once a crash costs a day.
Too many epochs on one stale batch
Loss keeps falling on the same batch while live return stalls, then falls.
Symptom: batch loss improves for twenty epochs while the next fresh batch scores worse, a classic overfit gap. Fix: hold epochs near three to ten, refresh the batch on schedule, and tune epoch count like a learning rate. Clipping bounds each step, not the sum of many small steps.
The Quick Version
- The ratio divides current action likelihood by the old likelihood, measuring movement from the collecting policy.
- The clipped objective keeps the smaller of the raw and band pinned terms, removing the reward for leaving when .
- Our six ratios at advantage 1.0 pay 0.50, 0.85, 1.00, 1.15, 1.20, and 1.20, so 2.5 earns nothing over 1.5.
- The flat roof lets one batch serve several epochs without running away.
- Reusing dear rollouts safely is why PPO carries human feedback training.
- Band width and epoch count stay small: clipping caps steps, not their sum.