LLM-as-a-Judge
When many answers can be right, a stronger model grades each output against a rubric or picks a winner head to head, giving you human-like signal at script speed.
Why Does This Exist?
Ask ten people to decline a job offer politely and you'll get ten different emails, all fine. That's the whole problem. When many answers can be right, string match has nothing to hold onto. A correct answer that uses different words still scores zero.
So teams fall back on human reading. Humans catch tone, correctness, and missing nuance. But you can't put a human in your pull request loop. Reading is slow, pricey, and two graders often disagree, which makes nightly regression checks painful.
LLM-as-a-judge fills that gap. You hand outputs to a stronger instruction tuned model with a written rubric and let it grade or compare at script speed. You get signal that reads more like a human than any string metric, cheap enough to run on every change.
This page covers the prompted judge: one model call, one rubric, no extra training. Its trained cousin is reward modeling, which fits a scorer on preference data for reinforcement learning. Votes that need a leaderboard go to pairwise preference and Elo. Judges applied to grounded answers belong to RAG evaluation, and judges used after a tune belong to post-training evaluation.
You need one idea first: exact match checks characters, human eval checks meaning. Exact match is fast and blind to paraphrase. Human eval sees meaning and costs time. A judge tries to sit between them.
Think of It Like This
A county fair pie contest
You enter a pie contest with twenty apple pies. No two look alike, and several deserve to win. The head judge cannot use a ruler. She uses a score sheet: crust, filling, appearance, each 1 to 5. For one pie she writes a number. For two pies side by side she just points at the better slice.
Here is the part that maps directly. The score sheet is your rubric. Scoring one pie alone is pointwise. Pointing at the better of two is pairwise. Her tasting notes come before the number, because a number without reasons drifts. And one bite never decides anything: she averages many tastes.
Where it stops: she actually tastes pie. Your judge never tastes truth. It matches language patterns, so a tall glossy pie can outscore a plain better tasting one. That is verbosity bias. Serve the same two pies in swapped order and her pick should not change. If your judge flips, you have position bias, not a verdict.
How It Actually Works
A judge call has four parts. Call the user prompt , in plain words the question you asked the model under test. Call its answer , the text you want graded. Call the rubric , the written sheet that says what counts as a 1, a 3, a 5. Call the judge , the stronger model doing the grading. The call sends plus plus to and parses back a verdict. Don't let answer first with the number. You ask for a short critique first, then the number, because text generated earlier shapes what comes next.
What the judge receives and returns
The prompt template is boring on purpose. It states the role, pastes , pastes , pastes , then says: write a brief critique that quotes the rubric, then on a new line write exactly Score: k. That last line exists so a regex can pull out the integer. If the format breaks, you retry or discard the call. No retry logic, no score.
Pointwise: one answer, one number
Pointwise means one pair in, one score out, with in . In plain words: grade this answer alone against the sheet. You then average over many items. If scores are through , with meaning the count of graded answers, the reported quality is the mean .
Pointwise tracks absolute quality over time, which is why teams like it for dashboards. Its weakness is drift. What counted as a 4 last month can read as a 3 today once the judge prompt or model shifts. Small rubric edits move the whole scale.
Pairwise: two answers, one winner
Pairwise means one prompt plus two answers and in, one verdict out, with in . In plain words: read both, pick the better one, ties allowed. You anonymize which system wrote which, and you randomize left and right placement.
One vote proves little. You aggregate. Count wins for A, wins for B, ties, with the total votes. The win rate for A is . In plain words: each tie counts as half a win. Pairwise is steadier than pointwise because comparing beats scoring. People and models both find relative calls easier than absolute ones. The cost is combinatorics: many systems need many matches, which is where Elo style aggregation earns its keep.
Worked example with real numbers
Take five pointwise scores . Sum them: . Divide by : . So the run reports 3.60. That is the number in the top lane of the diagram.
Now run nine pairwise votes: A wins 5, B wins 3, 1 tie. So , , , . Plug into the win rate: . A leads, but 0.61 on 9 votes is a nudge, not a coronation. That is the number in the bottom lane.
Trade offs, limits, and when to skip it
The trade is speed for correlated error. You buy fast repeatable signal and you pay in shared blind spots: the judge shares training, style, and gaps with the models it grades.
Don't use a judge for the final call on safety, medical, legal, or any release where a wrong grade hurts people. Don't use it where the judge lacks the facts, like fresh internal docs it never saw, unless you paste the reference into and grade groundedness only. Don't use the same model family to both compete and judge without saying so.
The assumption is correlation with humans: judge rankings track what your users would pick. When that fails, everything downstream fails quietly. Long fluent answers outscore short correct ones. The first shown answer gets a bump. A judge from one family prefers its own phrasing. You counter with structure: critique before the number, swapped positions averaged together, blinded identities, a rubric that penalizes fluff, and a human anchor you recheck whenever the judge or rubric changes.
Code
The snippet below does the only math a judge pipeline owns: averaging. Scores and votes come from the judge calls above. The point is aggregation, not prompting.
from collections import Counter
def mean_score(scores: list[float]) -> float: """Mean of pointwise rubric scores.""" return sum(scores) / len(scores)
def win_rate(votes: list[str]) -> float: """Share of pairwise votes for A, counting each tie as half a win.""" counts: Counter[str] = Counter(votes) wins: float = counts.get("A", 0) + 0.5 * counts.get("tie", 0) return wins / len(votes)
scores: list[float] = [4.0, 3.0, 5.0, 2.0, 4.0]votes: list[str] = ["A", "B", "A", "tie", "A", "B", "A", "B", "A"]print(f"{mean_score(scores):.2f}") # -> 3.60print(f"{win_rate(votes):.2f}") # -> 0.61Same 3.60 and 0.61 as the worked example. If your parser ever returns a different pair on these inputs, the bug is in parsing or counting, not in the rubric.
Watch Out For
Your winner flips when you swap A and B
Position bias is the classic judge failure. The judge prefers whichever answer it read first, or second, regardless of content. Symptom: A beats B 7 to 2 with A on the left, then loses 3 to 6 with A on the right. Your leaderboard measures layout, not quality.
Fix it structurally. Run every pair twice with positions swapped and average the two verdicts. Randomize placement per call, blind the model names, and log accuracy by position so drift shows up. If the swap changes the winner, don't ship the ranking. Tighten the rubric first.
You trust the judge with no human anchor
A judge without a human reference drifts into self-love. Symptom: scores look clean and climb each week, yet spot checks with users disagree, and outputs from the judge's own family mysteriously top the chart. Long answers keep winning even when reviewers call them padded.
Fix it with double grading. Take a shared sample you can afford to grade twice, have humans and the judge grade the same items, and measure agreement before you trust automation. Rewrite vague rubric lines into checkable ones, add a line that penalizes unverified filler, and re-anchor whenever you swap judge models or edit the prompt. A judge that never gets checked becomes a mirror, not a meter.
The Quick Version
- A judge model grades open ended outputs because string match cannot see paraphrase and human reading cannot run on every change.
- Pointwise asks for an absolute rubric score per answer and averages them, which tracks trends but drifts when the rubric or judge shifts.
- Pairwise asks which of two answers wins and aggregates to win rate, which is steadier because comparing is easier than scoring.
- Force critique before the number, blind identities, swap positions and average, or bias will write your leaderboard for you.
- A judge is a proxy with shared blind spots, so anchor it to human grades and skip it for final safety or factual calls it cannot verify.