Skip to content
AI360XpertBeta

RL from Verifiable Rewards (RLVR)

When an answer can be checked by a rule instead of a judge, train on the rule. RLVR pays full marks only when code runs or math matches, so the model must be right, not just sound right.

RLVR scores a group of sampled answers with a checkable rule instead of a learned judge, turning 1, 0, 0, 1 into advantages of plus 1, minus 1, minus 1, plus 1
RLVR scores a group of sampled answers with a checkable rule instead of a learned judge, turning 1, 0, 0, 1 into advantages of plus 1, minus 1, minus 1, plus 1

Why Does This Exist?

Preference training made assistants pleasant, but its judge is a learned model, and learned judges can be charmed. Hand it a proof with confident formatting and a wrong final line, and the reward model may still score it highly, because it looks like the good answers it trained on. Train against that judge long enough and you get reward hacking: fluent, polished, wrong.

Some tasks don't need a judge at all. Code either passes its tests or it doesn't. A math answer either matches or it misses. RLVR (reinforcement learning with verifiable rewards) trains on that fact: for these tasks it retires the learned scorer and pays reward from a small program that checks the answer. That signal can't be flattered. It can only be satisfied.

If the training loop itself is new to you, start with RLHF: a policy writes answers and a reward signal pushes it toward better ones. RLVR keeps that loop and swaps only the reward half. The pushing is usually done by GRPO, which appears in the worked example below, and the taxonomy around this choice lives in process rewards and outcome rewards.

Think of It Like This

A test graded with an answer key

You sit a math test. You fill two pages with rough work, then copy one final number into the box at the bottom. The grader never reads your pages. She lays a punched answer key over your paper, and your number either shows through the hole or it doesn't. Tick or cross. Your rough work still matters, but only through the number it leads you to: work that catches your own slips produces more ticks, so week by week you learn to check yourself.

Where it stops: a real teacher gives partial credit and can see you knew the method when your arithmetic slipped. RLVR's checker can't. Right method, right reasoning, wrong box, and the score is zero. That strictness is the whole method, and the first pitfall below is what it costs you.

How It Actually Works

One prompt, a group of tries, and a checker that plays no favorites. That is the whole machine; the rest is bookkeeping.

The verifier replaces the judge

Fix a prompt xx. Say it asks for an equation's solution inside <answer> tags. The policy πθ\pi_\theta (your model plus its current weights θ\theta) samples not one answer but a group of GG of them: y1,y2,…,yGy_1, y_2, \ldots, y_G. Here GG is the group size, just a count of tries per prompt.

An extractor EE (a few lines of string handling) pulls the boxed answer out of each try. A verifier VV (a program: a test runner, an exact-match check, a symbolic comparison) compares it against the gold answer y^\hat{y} stored with the prompt, and pays a binary reward: ri=1r_i = 1 on a match, ri=0r_i = 0 otherwise. Here rir_i is the score of try number ii, and full marks or nothing are its only settings.

Two labels matter. The reward is an outcome reward because it grades the final answer and never the working; it is verifiable because a program computes it and no model judges it. The full taxonomy is in process rewards and outcome rewards.

From rewards to an update

Raw 1s and 0s don't say how surprised the optimizer should be, so each try is scored against its own group. Let μ\mu be the group's mean reward (the plain average) and σ\sigma its standard deviation (how spread out the rewards are). The advantage of try ii is

Ai=ri−μσA_i = \frac{r_i - \mu}{\sigma}

which reads: how much better or worse this try was than its siblings, measured in units of spread.

The update then has the familiar policy-gradient shape:

J(θ)=E[ρiAi]−β KL(πθ∥πref)J(\theta) = \mathbb{E}[\rho_i A_i] - \beta\,\mathrm{KL}(\pi_\theta \Vert \pi_{\mathrm{ref}})

Symbol by symbol: ρi=πθ(yi∣x)/πold(yi∣x)\rho_i = \pi_\theta(y_i \mid x) / \pi_{\mathrm{old}}(y_i \mid x) is the probability ratio, how much likelier this answer is under the new weights than under the sampling weights πold\pi_{\mathrm{old}} that produced it. In log form that is log⁡πθ(yi∣x)\log \pi_\theta(y_i \mid x): how likely the model currently finds those exact tokens, the number the gradient actually pushes. πref\pi_{\mathrm{ref}} is the frozen model from before this training round, the anchor. KL\mathrm{KL} measures how far the policy has drifted from that anchor, and β\beta is the small weight on the leash. Tries with positive AiA_i get their probability raised; negative ones get lowered; the penalty keeps the whole move honest.

GRPO is the usual optimizer here because the group already supplies the baseline, so no separate value model is needed. Implementations add clipping so one batch of all-or-nothing rewards can't drag the policy across the room in a single step.

Worked example: four answers, one mean

Prompt xx asks for 6 times 7, with the final number inside <answer> tags. The gold answer is y^=42\hat{y} = 42. Group size G=4G = 4. The policy samples four tries:

  • y1y_1 thinks awhile, then boxes 42 in <answer> tags. The extractor returns 42. Match, so r1=1r_1 = 1.
  • y2y_2 boxes 41 the same way. Confident and wrong, so r2=0r_2 = 0.
  • y3y_3 writes out forty-two with no tags at all. The extractor finds nothing, and nothing to check means no match, so r3=0r_3 = 0.
  • y4y_4 boxes 42 again. Match, so r4=1r_4 = 1.

So the reward vector is r=[1,0,0,1]r = [1, 0, 0, 1]. The mean is μ=(1+0+0+1)/4=0.5\mu = (1 + 0 + 0 + 1)/4 = 0.5. The population variance is σ2=(0.25+0.25+0.25+0.25)/4=0.25\sigma^2 = (0.25 + 0.25 + 0.25 + 0.25)/4 = 0.25, so σ=0.5\sigma = 0.5. The advantages are

A=[1, −1, −1, 1]A = [1,\ -1,\ -1,\ 1]

since (1−0.5)/0.5=1(1 - 0.5)/0.5 = 1 and (0−0.5)/0.5=−1(0 - 0.5)/0.5 = -1. The update pushes the policy toward the two tries that boxed 42 and away from the confident 41 and the tagless forty-two.

Notice the tie the numbers force: both correct tries earn the same +1+1 whether the thinking behind them was elegant or clumsy. The verifier grades the box, never the path. That is the trade RLVR makes: a signal that can't be gamed, for a signal blind to style, effort, and near misses.

So reach for RLVR where answers check themselves: math with known answers, code with tests, formal tasks with validators. Don't reach for it where no checker exists: open-ended writing, tone, helpfulness, taste. There the RLHF loop and preference methods keep the job, and reasoning models that serve chat mix both: verifiable rewards for the checkable slice, human preference for the rest. The method assumes your verifier equals truth and your extractor reads every honest answer. The pitfalls are what breaks when either belief fails.

Code

The point of this snippet is the group scoring: binary rewards in, advantages out, on the same four numbers as the example.

def group_advantages(rewards: list[float]) -> list[float]:    n = len(rewards)    mean = sum(rewards) / n    var = sum((r - mean) ** 2 for r in rewards) / n  # population variance    std = var ** 0.5    if std == 0.0:        return [0.0 for _ in rewards]  # every sample agreed: nothing to learn    return [(r - mean) / std for r in rewards]

rewards: list[float] = [1.0, 0.0, 0.0, 1.0]print(group_advantages(rewards))  # -> [1.0, -1.0, -1.0, 1.0]

It prints [1.0, -1.0, -1.0, 1.0], the same advantages the hand computation gave. The zero-spread guard matters: hand it four identical scores and it returns zeros, which is the second pitfall below in code form.

Watch Out For

The right answer in the wrong box scores zero

The symptom is a flat reward curve while your own spot-checks find correct reasoning in the traces. The cause is extractor strictness: a missing tag, a trailing period, stray spaces, an unusual minus sign, or forty-two spelled out. The verifier never sees the reasoning, only what the extractor hands it, so good work in a bad box scores what wrong work scores.

Normalize before you compare: strip whitespace and trailing punctuation, and treat numeric equivalents as one answer. Pay a small separate format reward so the model learns the box without starving while it does, and chart the extraction-failure rate on its own. When that chart climbs as reward falls, your checker is failing, not your model.

Groups where everyone agrees teach nothing

The symptom is advantages hovering near zero while the policy barely moves for whole batches. The cause is agreement: all four samples pass or all four fail, so the mean equals every reward and each advantage computes to exactly zero. The update has nothing to push toward and nothing to push away from.

Keep prompt difficulty mixed so most groups hold both passes and fails, skip zero-spread groups in the update, and track the share of dead groups per batch. When that share climbs, your prompts drifted too easy or too hard. The algorithm didn't break; its food ran out.

The Quick Version

  • RLVR replaces the learned reward model with a checkable rule: code that passes its tests or math that matches the answer scores 1, and the rest scores 0.
  • Each prompt yields a small group of sampled answers, and every answer is scored against the group mean, so beating the group average is what gets reinforced.
  • The only way to score is to be actually right, so working that double-checks itself earns more reward over time, and long self-correcting chains grow out of that pressure.
  • RLVR fits only tasks with checkable truth; style, tone, and taste have no verifier, so preference training keeps those jobs.
  • Strict answer extraction silently zeroes correct work, and groups where every sample agrees carry zero advantage, so monitor extraction failures and dead groups from day one.