RLHF vs DPO vs GRPO
Comparing alignment techniques used to make LLMs helpful and safe.
Verdict: Use DPO as the robust, simple default for aligning models without needing a secondary reward model; use GRPO for highly reasoning-focused models to save massive memory costs.
The Short Answer
RLHF (Reinforcement Learning from Human Feedback) aligns models by training a separate "Reward Model" that judges the LLM, then using PPO to optimize the LLM against it. DPO (Direct Preference Optimization) completely eliminates the Reward Model by mathematically solving the RL problem directly on pairs of (chosen, rejected) text. GRPO (Group Relative Policy Optimization), popularized by DeepSeek, generates a group of answers for one prompt and scores them relative to each other, eliminating the need for a memory-heavy reference model.
Where They Differ
| Feature | RLHF | DPO | GRPO |
|---|---|---|---|
| Reward Model Required? | Yes | No | Yes (or rule-based) |
| Reference Model Required? | Yes | Yes | No |
| Memory Overhead | Extremely High (loads multiple models) | High (loads policy and reference) | Moderate |
| Training Complexity | Very Complex (PPO is unstable) | Simple (standard classification loss) | Moderate |
Choose DPO When
- You want the simplest alignment pipeline: DPO has effectively replaced RLHF in the open-source community because it turns alignment into a simple supervised learning task. If you have data shaped like
[Prompt, Good Answer, Bad Answer], you run DPO and the model aligns.
Choose GRPO When
- You are training models on math or coding: GRPO relies on relative rewards within a generated group. If a prompt generates 5 code snippets, and 2 pass unit tests, GRPO increases the likelihood of the passing ones relative to the failing ones. This is the exact mechanism DeepSeek-R1 used to achieve breakthrough reasoning performance.
- You are constrained by VRAM: PPO and DPO require you to load a frozen "Reference Model" in memory to prevent the model from drifting too far. GRPO uses the mean of the generated group as its own baseline, deleting the need to load the reference model.
What People Get Wrong
People assume RLHF is obsolete because DPO is mathematically cleaner. In reality, the absolute frontier models (like GPT-4 or Claude 3.5 Sonnet) still use proprietary variants of PPO/RLHF, because when you have infinite compute, training a dedicated, highly nuanced Reward Model can sometimes yield slightly better safety boundaries than pure DPO.