RL from Verifiable Rewards (RLVR)

Why Does This Exist?

In traditional RLHF, the Reward Model is a neural network. Because it is a statistical approximation, it can be tricked. If the Policy Model writes an elegantly formatted, highly confident, perfectly structured mathematical proof that concludes $2 + 2 = 5$, a neural Reward Model will often give it a high score because it looks like a good answer. The Policy Model learns to be a confident liar (Reward Hacking).

Reinforcement Learning from Verifiable Rewards (RLVR) solves this by firing the neural Reward Model entirely. For domains like math, code, and formal logic, we don't need a vibe check. We have deterministic truth. We can run the Python code. We can check the math equation. If the code compiles and passes the unit test, Reward = 1. If it fails, Reward = 0.

Think of It Like This

An art critic vs. a math teacher

Traditional RLHF is an art critic. You paint a picture, and the critic says "I like the colors" (high reward). You can easily paint something that appeals to the critic's specific biases, even if the anatomy of the subject is wrong.

RLVR is a math teacher with an answer key. The teacher doesn't care if your handwriting is beautiful or your essay is polite. If the final number in the box doesn't match the answer key, you fail. You cannot trick the answer key.

How It Actually Works

The Pipeline

1. The model is given a prompt with a known ground-truth answer (e.g., a LeetCode problem with hidden unit tests).
2. The model generates a response. To make it parsable, it is often instructed to format its output rigidly, such as <think>...</think><answer>...</answer>.
3. An automated script extracts the text inside the <answer> tag.
4. A deterministic engine (Python interpreter, regex matcher, or SymPy math engine) verifies the answer.
5. If it matches, the model gets a +1. If it fails, -1.
6. The model is updated using GRPO or PPO.

Emergent Reasoning

RLVR is the core breakthrough behind reasoning models like DeepSeek-R1 or OpenAI's o1. When a model is trained with RLVR for thousands of steps, a fascinating phenomenon occurs. Because the only way to get a reward is to be factually correct, the model learns that spending more time inside the <think> tag drastically increases its chances of getting the right answer.

It spontaneously learns to double-check its work, break problems into smaller steps, and recognize its own errors. No human explicitly taught it to do this in SFT; the behavior emerged purely because the rigid constraints of verifiable rewards demanded it.

Watch Out For

Domain Limitations

RLVR is a superpower for coding and mathematics. It is completely useless for open-ended creative writing, summarization, or tone alignment. You cannot write a unit test to verify if a poem is beautiful. For general-purpose assistants, RLVR must be combined with traditional RLHF or DPO.

The Quick Version

• RLVR replaces the neural Reward Model with a deterministic verifier (code execution, math solvers, regex).
• It completely eliminates Reward Hacking because the rules of the system cannot be tricked by confident-sounding text.
• It is the engine that drives modern reasoning models, causing behaviors like self-correction and chain-of-thought to emerge spontaneously.
• It only works for domains with objective, verifiable truth.

What to Read Next

• GRPO is the memory-efficient reinforcement learning algorithm usually paired with RLVR.
• Reward Modeling explains the exact fuzzy, neural system that RLVR was built to replace.