Gen AI

Reward Modeling

Before you can use reinforcement learning to train an LLM, you need a teacher to grade it. A reward model is an LLM trained to act as that teacher, reading text and outputting a single number representing how helpful or harmless the text is.

A reward model takes in a prompt and a response, and outputs a scalar score predicting human preference.

Why Does This Exist?

Reinforcement Learning from Human Feedback (RLHF) requires a model to be rewarded when it generates a good answer and penalized when it generates a bad one.

If you were training a robot to play chess, the reward is obvious: +1 for a win, -1 for a loss. But how do you programmatically score an essay on French history? You can't write a Python script to evaluate prose. You could have humans read every single generation and rate them, but humans take minutes to read an essay, and a reinforcement learning algorithm needs to generate and score tens of thousands of essays per second.

The Reward Model (RM) solves this bottleneck. It is a separate neural network (usually a cloned, smaller version of the LLM) that acts as an automated, high-speed human proxy.

Think of It Like This

Training a junior grader

Imagine a professor who wants to teach a massive class of 10,000 students. The professor cannot grade 10,000 essays a day (human bottleneck).

Instead, the professor spends a week grading 500 essays, carefully writing a rubric. They hand this to a Teaching Assistant (the Reward Model). The TA studies the 500 graded essays until they understand exactly what the professor likes and dislikes. From then on, the TA grades the 10,000 essays per day at lightning speed, allowing the students (the LLM) to get immediate feedback.

How It Actually Works

The Data: Pairwise Preferences

You don't train a reward model by asking humans to score essays out of 10. Humans are terrible at absolute scoring (my 7/10 might be your 9/10). Humans are excellent at comparisons.

The base LLM generates two different answers (Answer A and Answer B) to a single prompt.
A human annotator reads both and simply clicks "A is better than B".
This creates a dataset of pairwise preferences: $(x, y_w, y_l)$ where $x$ is the prompt, $y_w$ is the winning response, and $y_l$ is the losing response.

Training the RM

The Reward Model is initialized from a Supervised Fine-Tuned (SFT) model. The final layer (the language modeling head that predicts the next word) is removed and replaced with a linear layer that outputs a single scalar number.

During training, the RM is fed the prompt and the winning response, and it outputs a score: $R(x, y_w)$ . It is then fed the prompt and the losing response, outputting: $R(x, y_l)$ .

The loss function (typically based on the Bradley-Terry model of choice) penalizes the RM if $R(x, y_w)$ is not significantly higher than $R(x, y_l)$ . Over thousands of steps, the RM learns to assign higher numbers to text that exhibits the traits the humans preferred (e.g., lack of hallucinations, polite tone, clear formatting).

Reward Hacking

Because the RM is just a statistical approximation of human preference, it has blind spots. If the human annotators subconsciously preferred longer answers (even if they were rambling), the RM will learn that "length = high score."

When you attach this RM to an RL algorithm (like PPO), the RL algorithm will relentlessly optimize for the highest score. It will discover the RM's blind spot and start generating massive walls of useless text because the RM gives it a high score. This is called Reward Hacking, and it is the hardest part of alignment engineering.

Watch Out For

Using a weak reward model for a strong policy model

If you use a 3B parameter model to act as the Reward Model for a 70B parameter LLM, the 70B model will easily outsmart the grader. It will generate text that looks correct to the 3B model but is actually deeply flawed. The Reward Model must generally be as capable, or nearly as capable, as the model it is grading.

The Quick Version

A Reward Model is an LLM modified to output a single score instead of text.
It is trained on human preference data (Answer A is better than Answer B).
It acts as an automated, high-speed human proxy to provide rewards during RLHF.
It is highly susceptible to "reward hacking" if its training data contains hidden biases (like preferring long answers).