Rejection Sampling Fine-Tuning
Sample several answers per prompt, let a checker keep only the best one, and fine-tune on those winners. The model's everyday answer starts to look like its old lucky best.
Why Does This Exist?
Good demonstrations are the bottleneck in post-training. People write them slowly, they disagree with each other, and some skills need thousands of examples. Yet the model you're trying to teach can already produce a good answer some of the time. Rejection sampling fine-tuning (RFT) turns that occasional good answer into the training data: sample several responses, keep the best, and teach the model to produce that one every time.
Say one greedy answer to a coding prompt fails its tests, but one of 8 high-temperature samples passes. Judging which answer is best costs far less than writing the solution by hand. A test suite, a checker script, or a trained judge can score all 8 in seconds. You throw away 7 and fine-tune on the winner.
This page covers the sample, score, keep, and retrain loop. The scoring function itself lives in reward modeling. The final training step is plain supervised fine-tuning. The reinforcement learning alternative this loop lets you skip is RLHF.
The one idea you need first: supervised fine-tuning trains a model to imitate a set of example responses, token by token. RFT changes nothing about that step. It changes only where the examples come from: the model's own filtered outputs instead of human pens.
Think of It Like This
The high jumper and the video camera
You coach a high jumper who clears the bar once in every 8 attempts but can't tell you what the good jumps felt like. So you film 8 jumps, delete the 7 clips where the hips hit the bar, and make the athlete drill the one clearance on repeat. Within weeks the clearance becomes the normal jump, not the lucky one.
That maps part for part. The 8 filmed jumps are your samples. Watching the tape and deleting clips is the verifier scoring each response. The surviving clip is the kept best answer. Drilling it is supervised fine-tuning.
Where it stops: the camera never lies, but your verifier can. If your judge rewards height over form, drilling bakes that taste in. And the athlete can only drill a clearance they landed at least once. Nothing in this loop teaches a jump the model never makes.
How It Actually Works
For one prompt , you draw candidate responses through from the model at a high sampling temperature . Temperature is the knob that spreads the model's bets: low repeats the safe answer, high deals variety. A verifier scores each candidate. Verifier is the general name for whatever judges: a unit test, a proof checker, a trained reward model, or a stronger model acting as judge. You keep the top scorer and drop the rest. The kept pairs across many prompts form a dataset, and one ordinary SFT run trains the model on it.
The loop in four steps
- Collect prompts without answers. Coding tasks, math questions, anything with a checkable result work best, because there the checker is free and honest.
- Sample responses per prompt at high temperature. Eight is a common choice: large enough that a good answer usually shows up, small enough to afford.
- Score all with the verifier and keep the single best. The fraction you keep, here 1 of 8, is the acceptance rate.
- Fine-tune the original model on the kept pairs with the standard next-token loss. No policy gradients, no value network, no KL controller.
If you kept all answers and let them vote instead of training on the winner, that would be self-consistency. Same sampling instinct, different payoff: voting helps one answer, RFT improves every future answer.
Worked example: 8 answers, 1 keeper
One prompt, , verifier scores:
Candidate 2 wins at . The batch mean is , about . Filtering moves the training target from an average answer worth to a winner worth . The acceptance rate is 1 of 8, which is 12.5%: 7 generations in 8 are thrown away.
Scale that to 1000 prompts. You generate 8000 answers, score them, keep 1000 winners, and fine-tune on those. The model's plain single-sample answer starts to resemble its old best-of-8. That is the whole bet: today's luck becomes tomorrow's habit.
Why the average moves, and what it costs
Training only ever sees winners, so gradient steps pull probability toward responses shaped like . Each SFT step says: next time, put more weight here. Nothing about the model changed except which of its own habits get reinforced.
The price is compute, not people. You pay roughly times the generation cost plus scoring for every kept example. That trade beats human annotation when verification is cheap (tests, checkers) and loses when it isn't. So here is the when-not-to-use: with no trustworthy verifier, on open-ended writing with only a weak judge, this loop distills the judge's taste, flaws included. Then you're better off with human-written demonstrations or preference data from real raters, as covered in synthetic data for post-training.
Code
The point of this snippet is the filter itself, the three lines everything else depends on: pick the winner and measure what filtering bought you.
scores: list[float] = [0.12, 0.85, 0.31, 0.47, 0.22, 0.64, 0.38, 0.55]
def keep_best(scores: list[float]) -> tuple[int, float, float]: best: int = max(range(len(scores)), key=lambda i: scores[i]) mean_all: float = sum(scores) / len(scores) return best, scores[best], mean_all
best, kept, mean_all = keep_best(scores)print(best, kept, round(mean_all, 4)) # -> 1 0.85 0.4425print(f"{1 / len(scores):.1%}") # -> 12.5%The index is 0-based, so best of 1 means candidate 2, the 0.85. Filtering lifts the training target from 0.44 to 0.85 and keeps 1 in 8. The dataset, the SFT run, and the diagram above all flow from those three numbers.
Watch Out For
The verifier's blind spot becomes the model's personality
Suppose your judge gives long answers extra credit. RFT keeps the longest answers, SFT teaches them, and the next round's samples get longer still. Training loss falls. Kept scores rise. You feel great until a person reads the outputs and finds rambling.
The symptom: verifier scores climb across rounds while independent checks stay flat or drop. Those checks are a held-out judge, human spot reads, or task tests the filter never saw.
The fix: never grade with the same judge that filtered. Hold out a second judge or a human spot-check, blunt the exploitable proxy (for example, penalize length), and re-verify winners on tests the selection step never saw.
Winners-only training narrows the model
Keep only the top 1 of 8 and every gradient step rewards the same style. Single-sample quality rises, but variety dies: ask 8 times and you get near-identical answers, flaws included. Run the loop again on the narrowed model and it sharpens further. You can see it in the numbers when single-sample accuracy climbs while best-of-8 stalls.
The symptom: the 8 samples stop disagreeing, and their errors look like copies of each other.
The fix: mix winners with the original SFT data so the model keeps its range, run one or two rounds and then resample fresh from the updated model, and track best-of- accuracy, not just single-sample accuracy.
The Quick Version
- Rejection sampling fine-tuning samples answers per prompt, keeps the verifier's top pick, and fine-tunes on the winners.
- Verification is the load-bearing wall: tests and checkers work, while a weak judge's taste gets distilled, flaws included.
- Keeping 1 of 8 turns best-of-8 luck into the model's default single answer.
- It costs about times the generation plus scoring per kept example, and it skips reinforcement learning entirely.
- Grade with an independent check, never the filtering judge, and mix winners with base data so the model keeps its range.