Skip to content
AI360XpertBeta

GRPO

GRPO skips the extra value model PPO needs. It samples a small group of answers to one prompt and treats beating the group average as good.

GRPO turns four scored answers into advantages measured from their own mean of 5, so above average answers are reinforced with no value model
GRPO turns four scored answers into advantages measured from their own mean of 5, so above average answers are reinforced with no value model

Why Does This Exist?

RLHF with PPO keeps four heavy things in memory: the policy you train, a frozen reference that keeps it honest, a reward model that scores answers, and a value model that predicts the score before it arrives. That last one exists for a single job. Advantage, the number that says "do more of this" or "do less of this," is reward minus expectation. PPO learns that expectation with a whole extra network, plus its optimizer state, plus its training loop.

That's a lot of hardware for a subtraction. Group Relative Policy Optimization (GRPO) asks a shorter question. If you've already sampled several answers to the same prompt, why predict the average? Measure it. Score the group, take its mean as the baseline, and reinforce whatever beat the mean. One network leaves the loop, and with it the value loss, the value warmup, and the failure mode where a bad critic poisons good rewards.

This page covers that swap: how the group becomes the baseline, how the update stays stable with no critic, and where the trick breaks. For the full loop it plugs into, read RLHF. For the scorer that still grades each answer, read reward modeling. For the sibling method that skips RL entirely and learns straight from pairs, read DPO. The three-way trade is laid out in RLHF vs DPO vs GRPO.

The one idea you need first. An advantage is reward minus baseline. Positive means the answer beat expectations, so the update pushes its probability up. Negative means it fell short, so the update pushes it down. The baseline adds no new facts. It just centers the signal so "good" means "better than the alternatives sitting next to it." If you want the general form behind policy updates, start with policy gradients, then come back.

Think of It Like This

Grading on a curve, with four essays

One assignment, four essays. You have no time to predict each grade before reading. So you read all four, score them 2, 8, 4 and 6 out of 10, and average to 5. Then you tell the class: write more like the two above 5, less like the two below. Nobody needed a forecast. The group set its own bar.

Each piece maps. The assignment is the prompt. The four essays are the sampled group. The class average is the baseline. Points above or below it are the advantages. "More like this" is the policy update.

Where it stops: your classroom has no frozen copy of each student's old style holding them close, no rule that caps how far one lesson can move them, and no problem with crowning the best of a bad lot. GRPO needs all three. If every essay is weak, the curve still declares winners, and training on that signal teaches the wrong lesson. The machinery below exists to handle exactly that.

How It Actually Works

GRPO keeps the PPO-style update and deletes the critic. For each prompt it samples a group, centers the rewards on their own mean, scales by their own spread, and runs a clipped update with a leash to a frozen reference. Three steps, each with one formula.

Sample a group and score it

Write qq for the prompt in plain words. Write GG for the group size, meaning how many answers you sample for that one prompt. Write oio_i for the ii-th answer in the group, where ii runs from 1 to GG. Write πold\pi_{old} for the behavior policy, meaning the model weights that actually generated the group. Write rir_i for the reward of answer oio_i, coming from a learned scorer or a rule that checks the answer.

The baseline is the group mean, written mm:

m=1G∑i=1Grim = \frac{1}{G} \sum_{i=1}^{G} r_i

No prediction, no second network. Add the GG scores, divide by GG. With the running fixture r=[2,8,4,6]r = [2, 8, 4, 6] and G=4G = 4, that's (2+8+4+6)/4=20/4=5(2 + 8 + 4 + 6) / 4 = 20 / 4 = 5. The bar is 5 because these four answers averaged 5, not because any model guessed 5 in advance.

Turn scores into advantages

The raw gap, written gig_i, is each reward minus that mean:

gi=ri−mg_i = r_i - m

For the fixture: −3-3, +3+3, −1-1, +1+1. They sum to zero, always. That's the part newcomers miss. Within one group, someone must lose for someone to win. A group where every answer is decent still punishes its worst member.

Raw gaps swing with the reward scale, so GRPO standardizes. Write ss for the group spread, meaning the population standard deviation (divide by GG, not GG minus 1):

s=1G∑i=1G(ri−m)2s = \sqrt{\frac{1}{G} \sum_{i=1}^{G} (r_i - m)^2}

The advantage, written AiA_i, is the gap divided by the spread:

Ai=ri−msA_i = \frac{r_i - m}{s}

For the fixture the squared gaps are 9, 9, 1 and 1, which sum to 20. Divide by 4 to get variance 5, so s=5≈2.24s = \sqrt{5} \approx 2.24. The advantages are −3/2.24≈−1.34-3 / 2.24 \approx -1.34, +3/2.24≈+1.34+3 / 2.24 \approx +1.34, −1/2.24≈−0.45-1 / 2.24 \approx -0.45 and +1/2.24≈+0.45+1 / 2.24 \approx +0.45. Same ordering as the raw gaps, but now a hotly contested group and a blowout group speak at comparable volume. That's the whole normalization story.

Update without a value model

Write πθ\pi_\theta for the policy you're training, where θ\theta holds its weights. Write ρi\rho_i for the probability ratio, meaning how much likelier the new weights find answer oio_i than the weights that generated it:

ρi=πθ(oi∣q)πold(oi∣q)\rho_i = \frac{\pi_\theta(o_i \mid q)}{\pi_{old}(o_i \mid q)}

The update pushes ρi\rho_i up when AiA_i is positive and down when it's negative, but clips ρi\rho_i to the band [1−ϵ,1+ϵ][1 - \epsilon, 1 + \epsilon], where ϵ\epsilon (epsilon) is a small width you set. A single lucky group can't yank the model across the room. Then it subtracts β\beta times the KL distance from a frozen reference πref\pi_{ref}, where KL measures how far the new policy has drifted and β\beta (beta) sets how hard the leash pulls. Large β\beta holds you near where supervised tuning left you. Small β\beta lets RL roam, including into reward hacking.

What you don't compute matters as much. There's no value loss, no generalized advantage estimation, no critic warmup. The price moved elsewhere: you pay GG generations per prompt instead of one critic forward pass. That's the trade, stated plainly. Memory down, generation up.

When the rewards are checkable, a math answer that's right or wrong, code that passes its tests, the group signal is cleanest, because the scorer can't play favorites with style. That setting has its own name: RLVR. When rewards are pure taste, length and polish can win every group, and relative scoring amplifies the bias instead of removing it. If your data is only chosen-versus-rejected pairs with no groups at all, DPO is the simpler tool. Don't reach for GRPO when you have no trustworthy way to score a group.

Code

This snippet replays the fixture so you can check the arithmetic before trusting a trainer. It uses only the standard library.

import statistics

def grpo_advantages(rewards: list[float]) -> list[float]:    """Centered, standardized advantages for one prompt group."""    mean: float = statistics.fmean(rewards)    spread: float = statistics.pstdev(rewards)  # population std, divides by G    return [(r - mean) / spread for r in rewards]

rewards: list[float] = [2.0, 8.0, 4.0, 6.0]mean: float = statistics.fmean(rewards)gaps: list[float] = [r - mean for r in rewards]spread: float = statistics.pstdev(rewards)adv: list[float] = grpo_advantages(rewards)print(mean)  # -> 5.0print(gaps)  # -> [-3.0, 3.0, -1.0, 1.0]print(round(spread, 4))  # -> 2.2361print([round(a, 2) for a in adv])  # -> [-1.34, 1.34, -0.45, 0.45]

The prints match the hand calculation: mean 5, gaps minus 3, plus 3, minus 1, plus 1, spread about 2.24, advantages about minus 1.34, plus 1.34, minus 0.45, plus 0.45. Change the last reward to 5.0 and watch all four advantages shrink. That's the group setting its own bar.

Watch Out For

Every answer in the group scores the same

Symptom: training burns compute and learns nothing, or worse, throws divide-by-zero warnings. The rewards come back tied, say [6,6,6,6][6, 6, 6, 6], so the spread ss is 0 and every advantage is 0/00 / 0. Some code emits NaNs here and the NaNs ride the gradients into the weights.

Skip tied groups. Guard the update: when the spread is zero (or below a tiny floor), drop the group before dividing. Then track how often you skip. A high skip rate is telling you the prompt is too easy, too hard, or the checker is too coarse to rank answers. Fix the task mix or the checker, not the optimizer.

The group amplifies a biased judge

Symptom: longer, more polished answers win every group while correctness stalls. You inspect individual groups and each one looks fair, yet across a week of training the model gets wordier and no smarter. Relative scoring can't fix a scorer that loves style. It just reproduces the bias GG times per prompt.

Calibrate the judge before scaling the loop. Score a held-out sample by hand and check whether length predicts reward better than correctness does. Prefer checkable rewards wherever the task allows, keep the KL leash on, and watch the group mean over time. If the mean climbs while held-out quality doesn't, your reward signal is the problem.

The Quick Version

  • GRPO replaces PPO's learned value baseline with the mean reward of a small group sampled for the same prompt, so no critic network is trained.
  • The advantage is the centered, scaled gap Ai=(ri−m)/sA_i = (r_i - m) / s, where mm is the group mean and ss is the group spread.
  • The fixture gives rewards 2, 8, 4 and 6, mean 5, gaps minus 3, plus 3, minus 1, plus 1, and standardized advantages near minus 1.34, plus 1.34, minus 0.45, plus 0.45.
  • The policy update is a clipped probability ratio minus a KL leash to a frozen reference, with no value loss anywhere in the loop.
  • GRPO trades memory for generation: one fewer model to hold, GG answers to sample per prompt.
  • Tied groups carry zero signal and must be skipped, because their spread is zero and their advantages are undefined.
  • Relative scoring amplifies whatever the scorer likes, so biased or style-driven rewards get worse here, not better.