Skip to content
AI360XpertBeta

LLM-as-a-Judge

When many answers can be right, a stronger model grades each output against a rubric or picks a winner head to head, giving you human-like signal at script speed.

A judge model grades one answer against a rubric or picks a winner from two answers, then averages many calls into a score.
A judge model grades one answer against a rubric or picks a winner from two answers, then averages many calls into a score.

Why Does This Exist?

Ask ten people to decline a job offer politely and you'll get ten different emails, all fine. That's the whole problem. When many answers can be right, string match has nothing to hold onto. A correct answer that uses different words still scores zero.

So teams fall back on human reading. Humans catch tone, correctness, and missing nuance. But you can't put a human in your pull request loop. Reading is slow, pricey, and two graders often disagree, which makes nightly regression checks painful.

LLM-as-a-judge fills that gap. You hand outputs to a stronger instruction tuned model with a written rubric and let it grade or compare at script speed. You get signal that reads more like a human than any string metric, cheap enough to run on every change.

This page covers the prompted judge: one model call, one rubric, no extra training. Its trained cousin is reward modeling, which fits a scorer on preference data for reinforcement learning. Votes that need a leaderboard go to pairwise preference and Elo. Judges applied to grounded answers belong to RAG evaluation, and judges used after a tune belong to post-training evaluation.

You need one idea first: exact match checks characters, human eval checks meaning. Exact match is fast and blind to paraphrase. Human eval sees meaning and costs time. A judge tries to sit between them.

Think of It Like This

A county fair pie contest

You enter a pie contest with twenty apple pies. No two look alike, and several deserve to win. The head judge cannot use a ruler. She uses a score sheet: crust, filling, appearance, each 1 to 5. For one pie she writes a number. For two pies side by side she just points at the better slice.

Here is the part that maps directly. The score sheet is your rubric. Scoring one pie alone is pointwise. Pointing at the better of two is pairwise. Her tasting notes come before the number, because a number without reasons drifts. And one bite never decides anything: she averages many tastes.

Where it stops: she actually tastes pie. Your judge never tastes truth. It matches language patterns, so a tall glossy pie can outscore a plain better tasting one. That is verbosity bias. Serve the same two pies in swapped order and her pick should not change. If your judge flips, you have position bias, not a verdict.

How It Actually Works

A judge call has four parts. Call the user prompt xx, in plain words the question you asked the model under test. Call its answer yy, the text you want graded. Call the rubric rr, the written sheet that says what counts as a 1, a 3, a 5. Call the judge JJ, the stronger model doing the grading. The call sends xx plus yy plus rr to JJ and parses back a verdict. Don't let JJ answer first with the number. You ask for a short critique cc first, then the number, because text generated earlier shapes what comes next.

What the judge receives and returns

The prompt template is boring on purpose. It states the role, pastes xx, pastes yy, pastes rr, then says: write a brief critique cc that quotes the rubric, then on a new line write exactly Score: k. That last line exists so a regex can pull out the integer. If the format breaks, you retry or discard the call. No retry logic, no score.

Pointwise: one answer, one number

Pointwise means one (x,y)(x, y) pair in, one score ss out, with ss in {1,2,3,4,5}\{1, 2, 3, 4, 5\}. In plain words: grade this answer alone against the sheet. You then average over many items. If scores are s1s_1 through sNs_N, with NN meaning the count of graded answers, the reported quality is the mean (s1+⋯+sN)/N(s_1 + \cdots + s_N) / N.

Pointwise tracks absolute quality over time, which is why teams like it for dashboards. Its weakness is drift. What counted as a 4 last month can read as a 3 today once the judge prompt or model shifts. Small rubric edits move the whole scale.

Pairwise: two answers, one winner

Pairwise means one prompt xx plus two answers yAy_A and yBy_B in, one verdict vv out, with vv in {A,B,tie}\{A, B, \text{tie}\}. In plain words: read both, pick the better one, ties allowed. You anonymize which system wrote which, and you randomize left and right placement.

One vote proves little. You aggregate. Count NAN_A wins for A, NBN_B wins for B, NtieN_{\text{tie}} ties, with NN the total votes. The win rate for A is WA=(NA+0.5Ntie)/NW_A = (N_A + 0.5 N_{\text{tie}}) / N. In plain words: each tie counts as half a win. Pairwise is steadier than pointwise because comparing beats scoring. People and models both find relative calls easier than absolute ones. The cost is combinatorics: many systems need many matches, which is where Elo style aggregation earns its keep.

Worked example with real numbers

Take five pointwise scores [4,3,5,2,4][4, 3, 5, 2, 4]. Sum them: 4+3+5+2+4=184 + 3 + 5 + 2 + 4 = 18. Divide by N=5N = 5: 18/5=3.618 / 5 = 3.6. So the run reports 3.60. That is the number in the top lane of the diagram.

Now run nine pairwise votes: A wins 5, B wins 3, 1 tie. So NA=5N_A = 5, NB=3N_B = 3, Ntie=1N_{\text{tie}} = 1, N=9N = 9. Plug into the win rate: WA=(5+0.5×1)/9=5.5/9≈0.61W_A = (5 + 0.5 \times 1) / 9 = 5.5 / 9 \approx 0.61. A leads, but 0.61 on 9 votes is a nudge, not a coronation. That is the number in the bottom lane.

Trade offs, limits, and when to skip it

The trade is speed for correlated error. You buy fast repeatable signal and you pay in shared blind spots: the judge shares training, style, and gaps with the models it grades.

Don't use a judge for the final call on safety, medical, legal, or any release where a wrong grade hurts people. Don't use it where the judge lacks the facts, like fresh internal docs it never saw, unless you paste the reference into rr and grade groundedness only. Don't use the same model family to both compete and judge without saying so.

The assumption is correlation with humans: judge rankings track what your users would pick. When that fails, everything downstream fails quietly. Long fluent answers outscore short correct ones. The first shown answer gets a bump. A judge from one family prefers its own phrasing. You counter with structure: critique before the number, swapped positions averaged together, blinded identities, a rubric that penalizes fluff, and a human anchor you recheck whenever the judge or rubric changes.

Code

The snippet below does the only math a judge pipeline owns: averaging. Scores and votes come from the judge calls above. The point is aggregation, not prompting.

from collections import Counter

def mean_score(scores: list[float]) -> float:    """Mean of pointwise rubric scores."""    return sum(scores) / len(scores)

def win_rate(votes: list[str]) -> float:    """Share of pairwise votes for A, counting each tie as half a win."""    counts: Counter[str] = Counter(votes)    wins: float = counts.get("A", 0) + 0.5 * counts.get("tie", 0)    return wins / len(votes)

scores: list[float] = [4.0, 3.0, 5.0, 2.0, 4.0]votes: list[str] = ["A", "B", "A", "tie", "A", "B", "A", "B", "A"]print(f"{mean_score(scores):.2f}")  # -> 3.60print(f"{win_rate(votes):.2f}")  # -> 0.61

Same 3.60 and 0.61 as the worked example. If your parser ever returns a different pair on these inputs, the bug is in parsing or counting, not in the rubric.

Watch Out For

Your winner flips when you swap A and B

Position bias is the classic judge failure. The judge prefers whichever answer it read first, or second, regardless of content. Symptom: A beats B 7 to 2 with A on the left, then loses 3 to 6 with A on the right. Your leaderboard measures layout, not quality.

Fix it structurally. Run every pair twice with positions swapped and average the two verdicts. Randomize placement per call, blind the model names, and log accuracy by position so drift shows up. If the swap changes the winner, don't ship the ranking. Tighten the rubric first.

You trust the judge with no human anchor

A judge without a human reference drifts into self-love. Symptom: scores look clean and climb each week, yet spot checks with users disagree, and outputs from the judge's own family mysteriously top the chart. Long answers keep winning even when reviewers call them padded.

Fix it with double grading. Take a shared sample you can afford to grade twice, have humans and the judge grade the same items, and measure agreement before you trust automation. Rewrite vague rubric lines into checkable ones, add a line that penalizes unverified filler, and re-anchor whenever you swap judge models or edit the prompt. A judge that never gets checked becomes a mirror, not a meter.

The Quick Version

  • A judge model grades open ended outputs because string match cannot see paraphrase and human reading cannot run on every change.
  • Pointwise asks for an absolute rubric score per answer and averages them, which tracks trends but drifts when the rubric or judge shifts.
  • Pairwise asks which of two answers wins and aggregates to win rate, which is steadier because comparing is easier than scoring.
  • Force critique before the number, blind identities, swap positions and average, or bias will write your leaderboard for you.
  • A judge is a proxy with shared blind spots, so anchor it to human grades and skip it for final safety or factual calls it cannot verify.