Self-Rewarding Language Models

The Problem

Before this paper, aligning large language models typically relied on a frozen reward model trained from human preference data. This approach was bottlenecked by human performance and cost: humans make mistakes, and collecting large volumes of high-quality preference data is expensive. Furthermore, because the reward model is frozen during the reinforcement learning phase, it cannot improve its ability to judge responses even as the main language model gets better.

The Idea

The authors proposed Self-Rewarding Language Models, where the LLM itself is used to provide its own rewards during training via LLM-as-a-Judge prompting. Instead of relying on a static, human-trained external reward model, the model evaluates its own generated responses to create preference pairs. These pairs are then used to update the model using Direct Preference Optimization (DPO). As the model trains, both its generation capability and its reward-judging capability improve simultaneously.

How It Works

Self-Instruction Creation
The model generates new prompts and candidate responses based on a seed set of instructions.

LLM-as-a-Judge Evaluation
The model evaluates its own candidate responses using an LLM-as-a-Judge prompt, outputting a reward score (e.g., from 1 to 5) for each response.

Iterative Direct Preference Optimization (DPO)
The responses are ranked based on the model's self-generated rewards to form preference pairs (chosen and rejected). The model is then trained on these pairs using DPO.

Continual Self-Improvement
Because the same model acts as both the generator and the judge, updating the model improves both its instruction-following ability and its evaluation ability, allowing for multiple iterations of self-improvement.

Why It Mattered

By removing the reliance on a frozen, human-trained reward model, this approach demonstrated a path toward models that can continually improve beyond human capability levels. Fine-tuning Llama 2 70B on just three iterations of this method resulted in a model that outperformed major existing systems (like Claude 2, Gemini Pro, and GPT-4 0613) on the AlpacaEval 2.0 leaderboard, proving that an LLM can provide a sufficient training signal to improve itself.

What Came After

This paper accelerated the adoption of synthetic preference data and self-play paradigms in LLM alignment. It directly influenced later research into continuous self-improvement frameworks, process-based self-rewarding models, and methods that use models to generate training signals for advanced reasoning tasks, reducing the dependency on expensive human annotation.