Gen AI

Rejection Sampling Fine-Tuning

Instead of relying on humans to write perfect training data, you ask a model to answer a question 10 times. You use a reward model to pick the best answer, and then fine-tune the original model on its own best work.

Rejection sampling generates multiple candidate responses, uses a reward model to select the highest quality one, and uses that pair to supervise the model.

Why Does This Exist?

Supervised Fine-Tuning (SFT) requires tens of thousands of perfect, human-written examples. Humans are slow, expensive, and often inconsistent.

We know that Large Language Models are capable of generating brilliant answers, but they are also capable of generating mediocre ones. If you ask an LLM a complex math question, it might get it wrong on the first try, but if you ask it to generate 10 different attempts (by turning up the sampling temperature), one of those attempts is usually correct and beautifully reasoned.

Rejection Sampling Fine-Tuning (RSFT or RFT) exploits this asymmetry. It is much easier to verify a good answer than it is to generate one. We can use code execution, a stronger LLM, or a dedicated Reward Model to score the 10 attempts, throw away the 9 bad ones (rejection), and keep the best one. We then fine-tune the model on the best examples it generated.

Think of It Like This

The student who needs to trust their best instincts

Imagine a student who is a brilliant writer but lacks confidence. When given an essay prompt, they write five drafts. Four are messy, but one is a masterpiece.

A teacher (the reward model) reads all five, throws four in the trash, and hands the masterpiece back to the student. "Study this one," the teacher says. "This is how you should write every time."

By studying their own best work, the student eventually learns to produce the masterpiece on the first try.

How It Actually Works

The Pipeline

Prompt Generation: Collect a large set of prompts (e.g., coding challenges, logic puzzles) without answers.
Generation (Best-of-N): Pass each prompt to the target LLM and ask it to generate $N$ candidate responses (e.g., $N=8$ or $N=16$ ) at a high temperature to ensure variety.
Scoring/Verification: Evaluate all $N$ responses. For math or code, this is easy: run the code or check the final equation. For open-ended text, use an automated Reward Model (or a stronger LLM like GPT-4) to score the responses on helpfulness and accuracy.
Selection: For each prompt, keep only the highest-scoring response. Reject the rest.
Supervised Fine-Tuning: Take the original model and perform standard SFT using the new dataset of "Best-of-N" responses.

Why It Works

Models often suffer from a gap between what they know and what they output. The pretraining phase embeds deep logic, but the model doesn't always know how to traverse that logic on a single greedy decoding pass. Rejection sampling forces the model to explore its own latent capabilities. By fine-tuning on the successful explorations, you shift the probability distribution so that the model's "average" output becomes as good as its previous "best-out-of-10" output.

RFT vs. RLHF

Rejection Sampling is often used as a cheaper, more stable alternative to Reinforcement Learning from Human Feedback (RLHF) or Proximal Policy Optimization (PPO). PPO is notoriously difficult to tune and prone to collapsing. Rejection sampling distills the preference optimization directly back into a standard, highly stable SFT run. Llama 2 and Llama 3 famously used massive amounts of rejection sampling in their post-training pipelines.

Watch Out For

Reward Model Exploitation

If the evaluator (the Reward Model) has a blind spot—for example, it thinks longer answers are always better, even if they are rambling—the rejection sampling process will select the longest, most rambling answers. The model will then be fine-tuned to ramble. The quality of RFT is strictly capped by the quality of the verifier.

The Quick Version

Rejection Sampling Fine-Tuning (RFT) asks a model to generate $N$ answers to a single prompt.
A verifier (code execution or a reward model) scores the answers and selects the best one.
The model is then fine-tuned via SFT on its own best answers.
It is a highly stable, scalable way to improve reasoning and alignment without human annotators or complex RL algorithms.