Group Relative Policy Optimization (GRPO)

The Problem

In Reinforcement Learning from Human Feedback (RLHF), Proximal Policy Optimization (PPO) was the dominant algorithm. However, PPO requires a separate Critic model (or value model) to establish a baseline for how good a particular state is. This Critic model is typically the same size as the main Actor model (the LLM being trained). Training with PPO thus requires keeping multiple massive models in memory simultaneously (the Actor, the Critic, a reference model, and the reward model). This memory overhead created a massive compute bottleneck for scaling RLHF to larger models.

The Idea

The authors realized that instead of using a separate Critic model to estimate the baseline value of a prompt, they could estimate the baseline directly from the model's own generations. By generating a group of outputs for the same prompt and comparing their rewards, the model can inherently learn which responses are better or worse relative to its current average performance.

How It Works

GRPO drastically simplifies the PPO architecture by completely removing the Critic model.

Group Generation For a given prompt, the Actor model generates a group of $G$ distinct outputs (for example, 4 or 8 different responses).

Reward Scoring A separate Reward Model (or a rule-based checker, which is especially useful for math or code where correctness is objective) scores every output in the group. This gives a set of rewards $r_1, r_2, \dots, r_G$.

Relative Advantage Calculation Instead of a Critic predicting a baseline value to subtract from the reward, GRPO computes the mean ($\mu$) and standard deviation ($\sigma$) of the $G$ rewards. The advantage $A_i$ for each output is simply its standardized score within the group: $A_i = \frac{r_i - \mu}{\sigma}$

Policy Update The Actor model is then updated to increase the probability of generating outputs that scored above the group average (positive advantage) and decrease the probability of those that scored below (negative advantage). Like PPO, GRPO also uses a clipping mechanism and a KL-divergence penalty to ensure the policy doesn't drift too far from the reference model in a single step.

Why It Mattered

GRPO mathematically eliminated one of the largest memory bottlenecks in RL training. By dropping the Critic model, GRPO reduced the memory footprint of RLHF by nearly half. This allowed researchers to perform reinforcement learning on much larger models using the same hardware budget.

What Came After

GRPO was introduced in the DeepSeekMath paper to train a 7B model to achieve state-of-the-art mathematical reasoning. It later became the foundational RL algorithm for DeepSeek's larger models, including DeepSeek-Coder and DeepSeek-V3, proving that open-weight models could match or exceed proprietary models in reasoning tasks using vastly more efficient training pipelines.