Process vs Outcome Rewards
Grading only the final answer teaches a model to guess. Grading each reasoning step teaches it to reason.
Why Does This Exist?
Once a reasoning model can write out its steps, you have to decide what to grade. The cheapest option is the final answer. It matches, it scores. That is an outcome reward model (ORM).
It works until it teaches the wrong lesson. A model can write shaky steps, contradict itself mid-chain, and still land on the right final token by luck. The ORM sees the match and pays full marks. Train on that signal long enough and you get a confident guesser, not a careful reasoner. That failure has its own name: reward hacking.
A process reward model (PRM) grades the chain instead. It scores each step given the steps before it, so the exact step where the logic breaks gets flagged even when the final answer looks fine.
This page covers that split: what each scorer sees, how the scores combine, and when the extra step labels are worth paying for. For the scorer itself, start with reward modeling. For the training loop that consumes these scores, read GRPO. For the inference trick where a PRM picks the best of many tries, read test-time compute.
Think of It Like This
The exam with no partial credit, and the one with it
One teacher grades only the circled letter on your answer sheet. You guess C, C happens to be right, you get full marks. Your scratch page could say anything. The grade cannot tell guessing from knowing, so next time you guess again.
A second teacher grades the scratch page line by line. The first two lines earn ticks. The third line invents a rule that is not true, and that line gets a cross on the spot, even though your final circle still says C. You quickly learn that only sound lines earn points.
Where it stops: real exam lines are either right or wrong to a teacher who knows the subject. A PRM is a learned guess about correctness, not a teacher. When its step labels are noisy, it can punish good reasoning and reward fluent nonsense, which is exactly the failure the second half of this page is about.
How It Actually Works
Write for the prompt in plain words. Write the model's trace as steps followed by a final answer . Write for the reference answer you check against.
An outcome scorer looks only at the ending:
In words: (the outcome reward) is 1 when the final answer matches and 0 otherwise. The steps through do not appear in the formula. That absence is the whole weakness. Credit assignment is blind. Every step in a lucky trace gets reinforced, including the broken one.
A process scorer grades each step in context. Write for the score of step given the prompt and everything before it, a number between 0 and 1 that you read as the chance this step is sound:
One trace then yields a list . To pick or reinforce traces you need one number per trace, so you aggregate. Two common choices are the minimum, which asks whether every step holds, and the mean, which asks how the chain does on average:
The minimum is strict and popular for reranking, because one broken step should sink a proof. The mean is forgiving and smoother for training, because it still separates mostly-good chains from mostly-bad ones.
Worked example with one lucky trace
Prompt : compute . Reference : . The model writes three steps and a final:
- : split 27 into 20 and 7, so add . Sound. Say .
- : . Correct. Say .
- : . Wrong, the true sum is . Say .
- Final : . It matches , so .
The outcome check pays full marks: . The process check does not. Its minimum is , which points straight at . Its mean is , a mediocre score for a chain with one broken link. Same trace, opposite verdicts. That gap is the diagram above.
The trade in one paragraph
Step labels buy credit assignment at the price of supervision. Someone or something must judge every step, which costs far more than checking final answers, and those judgments assume steps segment cleanly into right-or-wrong units. When finals are cheap to check and chains are short, skip the machinery: an ORM plus sampling catches most of it. Reach for a PRM when chains are long, a single early slip poisons everything after it, or you need a verifier that rejects fluent nonsense at inference time.
Code
This snippet replays the fixture numbers so you can check the arithmetic. It uses only the standard library.
def outcome_reward(final: str, truth: str) -> float: """1.0 when the final answer matches, else 0.0. Steps are ignored.""" return 1.0 if final.strip() == truth.strip() else 0.0
def process_aggregate(step_scores: list[float], mode: str = "min") -> float: """Combine per-step soundness into one trace score.""" if mode == "min": return min(step_scores) return sum(step_scores) / len(step_scores)
scores: list[float] = [0.95, 0.92, 0.08]print(outcome_reward("42", "42")) # -> 1.0print(process_aggregate(scores, "min")) # -> 0.08print(round(process_aggregate(scores, "mean"), 2)) # -> 0.65The prints match the hand calculation: the outcome check says , the strict process aggregate says , the forgiving one says . Change the last score to and watch the minimum jump to . That sensitivity is why the minimum is such a sharp filter for reranking.
Watch Out For
Eval climbs while the reasoning rots
Symptom: final-answer accuracy rises for weeks, then you read twenty traces and half of them argue nonsense into correct finals. That is the ORM signature. The metric you optimized stopped measuring what you wanted.
Audit traces, not just finals. Sample chains every run and read them. Add a PRM as a reranker at inference, or mix a small set of step-checked prompts into training so lucky chains stop earning full pay.
Noisy step labels teach the PRM the wrong lesson
Symptom: the PRM gives high scores to smooth but false steps, or punishes unusual-but-valid shortcuts. Your reranker then picks polished wrong answers over plain right ones, and training drifts toward style instead of substance.
Calibrate before you scale. Score a held-out batch by hand, check whether the PRM agrees with you step by step, and distrust labels from weak judges on hard problems. When labels are thin, use the PRM only to veto obvious breaks rather than to rank fine differences.
The Quick Version
- An outcome reward grades only the final answer, so a flawed chain with a lucky final still scores full marks.
- A process reward scores each step given the steps before it, and the minimum across steps points at the exact faulty link.
- The fixture trace scores outcome 1 against a process minimum of 0.08 and a mean of 0.65, computed as 1.95 divided by 3.
- Step labels cost far more than final-answer checks, so skip PRMs when chains are short and finals are cheap to verify.
- An ORM trained too long breeds reward hacking: accuracy climbs while the reasoning underneath rots, so audit traces by hand.
- A PRM is only as honest as its step labels, and noisy labels turn it into a style judge rather than a logic judge.