Gen AI

Process vs Outcome Rewards

When training a model via reinforcement learning, grading only the final answer (Outcome) teaches the model to cheat. Grading every single intermediate step (Process) teaches the model true, rigorous logic.

Outcome reward models (ORM) only grade the final answer, while Process reward models (PRM) grade every individual step of the reasoning chain

Why Does This Exist?

After a reasoning model undergoes basic training, it goes through a Reinforcement Learning (RL) phase. In RL, the model attempts a problem, a "Reward Model" grades the attempt, and the main model updates its weights to get a higher grade next time.

Historically, the industry used Outcome Reward Models (ORMs). The ORM looks at the final answer and gives a simple thumbs up or thumbs down. If the math problem is $10 + 5$ , and the model outputs "15", it gets a reward.

The problem? ORMs encourage reward hacking. An LLM might generate this reasoning chain: "10 is an even number. 5 is odd. Even + Odd = 15. Therefore, the answer is 15." The logic is complete nonsense, but the final answer is correct. The ORM blindly gives a massive reward, accidentally teaching the model that hallucinated nonsense is a valid mathematical strategy.

Process Reward Models (PRMs) were invented to fix this.

Think of It Like This

Grading a high school math test

An Outcome Reward Model is a lazy teacher who only looks at the bubbled-in scantron sheet. If a student guesses 'C' and gets it right, they get an A+, even if they used entirely fabricated formulas on their scratch paper.

A Process Reward Model is a diligent teacher who demands you "show your work." They go through the scratch paper line by line. If you make a brilliant logical deduction in step 1, you get points. If you invent a fake formula in step 2, you get heavily penalized immediately, even if a subsequent math error accidentally results in the correct final answer.

How It Actually Works

The Outcome Reward Model (ORM)

ORMs are cheap to train and easy to run. You just need a dataset of questions and their final answers. During training, the generator model creates a full chain of thought and an answer. The ORM strips away the chain of thought, looks at the final answer, compares it to the dataset, and issues a 1.0 (correct) or -1.0 (incorrect). The generator model has to guess which part of its massive chain of thought was responsible for the grade.

The Process Reward Model (PRM)

PRMs are incredibly expensive to build. You must hire human experts to read thousands of AI-generated reasoning chains and manually grade every single sentence as correct, neutral, or incorrect.

Once trained, the PRM evaluates the generator model step-by-step.

Step 1: Valid assumption. (+0.5 reward)
Step 2: Correct calculation. (+0.5 reward)
Step 3: Logical leap / hallucination. (-1.0 reward)

At the exact moment the logic fails (Step 3), the PRM halts the reward. The generator model learns exactly which specific thought derailed the process, allowing it to finely tune its internal logic pathways.

The Verification Superpower

Beyond training, PRMs are the secret weapon for Test-Time Compute. When you generate 64 possible answers to a problem at inference time, you use a PRM to grade them. Because the PRM checks the logic line-by-line, it is highly resistant to picking a confidently-stated hallucination, guaranteeing the final output is logically sound.

Show Me the Code

This conceptually highlights how an ORM and a PRM grade the exact same hallucinated output.

1# The model's generated reasoning chain2reasoning_steps = [3    "The user wants 12 * 12.",4    "10 * 10 is 100.",5    "2 * 2 is 44.", # HALLUCINATION / MATH ERROR6    "100 + 44 = 144.",7    "The final answer is 144."8]9
10def ORM_evaluation(steps, expected_answer):11    # ORM only looks at the final conclusion12    final_output = steps[-1]13    if expected_answer in final_output:14        return "PASS: 1.0 (Reward Granted)"15    return "FAIL: 0.0"16
17def PRM_evaluation(steps):18    # PRM grades line-by-line using its own internal logic capabilities19    grades = []20    for step in steps:21        if "44" in step and "2 * 2" in step:22            grades.append("FAIL (-1.0) -> Logic broken here.")23            break # Halt grading on failure24        else:25            grades.append("PASS (+1.0)")26            27    return grades28
29print("--- Outcome Reward Model ---")30print(ORM_evaluation(reasoning_steps, "144"))31# -> PASS: 1.0 (Reward Granted)  <-- DISASTER! The ORM rewarded bad math!32
33print("\n--- Process Reward Model ---")34for i, grade in enumerate(PRM_evaluation(reasoning_steps)):35    print(f"Step {i+1}: {grade}")36# -> Step 1: PASS (+1.0)37# -> Step 2: PASS (+1.0)38# -> Step 3: FAIL (-1.0) -> Logic broken here.

The PRM successfully identified the hallucination in Step 3 and penalized the model, preventing it from learning that 2 * 2 = 44 is an acceptable path to a correct answer.

Watch Out For

The cost of PRMs

Building a PRM requires millions of human annotations from domain experts (PhDs in math, physics, coding). Unlike standard RLHF where anyone can rate which poem sounds better, grading complex logic requires deep expertise. This makes PRMs one of the most expensive and closely guarded intellectual properties of frontier AI labs (like OpenAI and Google).

The Quick Version

Outcome Reward Models (ORMs) grade an AI solely on whether its final answer is correct.
ORMs accidentally encourage reward hacking, where the model uses hallucinated, broken logic to accidentally arrive at the right answer.
Process Reward Models (PRMs) grade an AI line-by-line, evaluating the integrity of the intermediate reasoning steps.
PRMs teach the model true, rigorous logic and precisely identify where the model's thought process breaks down.
PRMs are also used during inference (test-time compute) as highly accurate verifiers to select the best answer from a pool of candidates.