GRPO

Why Does This Exist?

Standard Reinforcement Learning from Human Feedback (RLHF) uses an algorithm called PPO. To run PPO, you need four models in memory simultaneously:

1. The Policy Model (the one being trained).
2. The Reference Model (frozen, to prevent the policy from wandering too far).
3. The Reward Model (to score the answers).
4. The Value Model (a massive network that tries to predict what the Reward Model is going to say, used to calculate 'advantage').

If you are training a 70B parameter model, holding four 70B models in VRAM is financially ruinous. Group Relative Policy Optimization (GRPO), pioneered heavily in DeepSeekMath and DeepSeek-R1, elegantly solves this memory bottleneck by completely eliminating the Value Model.

Think of It Like This

Grading on a curve

In standard PPO, a student writes an essay. The TA (Value Model) guesses the essay will get a 70. The Professor (Reward Model) actually gives it an 80. The student is reinforced positively because they did +10 points better than expected.

In GRPO, there is no TA. The student simply writes 5 different essays. The Professor grades all five (e.g., 60, 70, 70, 80, 90). The average is 74. The student is reinforced positively on the essays that scored above 74, and penalized on the ones below. Grading on a curve removes the need for someone to predict the score beforehand.

How It Actually Works

The Advantage Calculation

In RL, Advantage is the metric that tells the model whether an action was good or bad. Positive advantage means "do this more," negative means "do this less." In PPO, Advantage = Actual Reward - Predicted Value (from the Value Model).

GRPO completely bypasses the Value Model by using a relative baseline.

1. The Policy Model generates a group of $G$ responses (e.g., 8 different answers to a single prompt).
2. The Reward Model (or a rule-based checker) scores all 8 responses.
3. The scores are normalized (mean-centered). The average score of the group becomes the baseline.
4. Advantage = Score - Group Average.

If an answer scored higher than the group's average, it gets a positive advantage. If it scored lower, it gets a negative advantage. The model is updated accordingly.

The Memory Savings

By eliminating the Value Model, GRPO immediately cuts the VRAM requirement of RLHF by roughly 25-30% (or more, if optimizer states are considered). This allows teams to train much larger models on the same hardware, or use significantly larger batch sizes and context windows during RL, which is critical for complex reasoning tasks.

Watch Out For

Small group sizes

The mathematics of GRPO depend on a statistically meaningful average. If you set the group size ($G$) to 2, the "average" is extremely noisy and the gradients will be unstable. A typical group size is 4 to 16. However, generating 16 responses per prompt increases the generation time during training. It trades VRAM savings for compute time.

The Quick Version

• Standard PPO requires a massive Value Model to calculate baseline expectations.
• GRPO eliminates the Value Model to save massive amounts of VRAM.
• It calculates Advantage by generating a group of responses and comparing each score against the group's average.
• It trades memory for generation compute (since it must generate $G$ responses per step) and is highly favored for training modern reasoning models.

What to Read Next

• RLVR explores how GRPO is used with verifiable math rules instead of fuzzy LLM reward models.
• DPO is a different approach that eliminates both the Value Model and the Reward Model entirely.