Skip to content
AI360XpertBeta

Prompt Regression Testing

Every prompt edit is a new deployment. One fixed case can break two working ones, so you rerun a frozen set of old cases and count what got fixed against what regressed.

Fixing 3 failing cases while breaking 2 passing ones lifts the headline from 85.0 to 87.5 percent, so the suite blocks the ship
Fixing 3 failing cases while breaking 2 passing ones lifts the headline from 85.0 to 87.5 percent, so the suite blocks the ship

Why Does This Exist?

You fix the refund answer on Monday. By Wednesday, tickets show the address-change answer broke. Nothing else shipped. The only change was one sentence in the system prompt.

That's normal for prompts, not bad luck. A prompt is one shared instruction read on every request, so a line that helps one case shifts how the model reads all the others. You can't patch case B and assume case A stayed put. Every prompt edit is a new deployment, and it needs the same proof a deployment needs: run the old cases again and show they still pass.

This page covers that proof: the frozen case set, the head-to-head run, and the fixed-versus-regressed count that gates the ship. It assumes you already score outputs somehow. If that part is new, start with LLM-as-a-judge, then come back. The search loop that hunts for better prompts on top of this harness lives in automatic prompt optimization.

Think of It Like This

The soup pot and its forty regulars

You run a lunch counter with one big pot of soup and forty regulars. One of them says the soup is bland, so you add salt. He's happy. But three regulars who liked it mild quietly stop coming, and you never hear why.

So you change the routine. Before any recipe change goes live, the same forty regulars taste the new pot and vote the way they always have. You count the changed minds in both directions: how many haters converted, and how many lovers you lost. The new recipe ships only if the second number is zero.

Where it stops: soup has one flavor, and your regulars vote with their mouths. A prompt has dozens of behaviors, and your cases vote through a scorer that can drift. The counting idea transfers exactly. The tasting does not run itself.

How It Actually Works

You keep a golden set: a frozen list of past inputs with the answers you demand, versioned in git next to the prompt. A candidate prompt never replaces the live one by argument. It earns the slot by beating the current prompt on that set without breaking anything that already works.

Concretely, the harness runs both prompts over the same cases with the same scorer pinned to the same version, then buckets every case by its pair of verdicts. Three buckets, no overlap.

The three buckets

For each case ii, write bib_i for the baseline verdict (11 for pass, 00 for fail) and nin_i for the candidate verdict on that same case. Let NN be the number of cases in the golden set. Then:

  • fixed: bi=0b_i = 0 and ni=1n_i = 1. Count them as FF.
  • regressed: bi=1b_i = 1 and ni=0n_i = 0. Count them as RR.
  • unchanged: the two verdicts agree. Count them as U=N−F−RU = N - F - R.

Headline pass rates are P1P_1 for the baseline and P2P_2 for the candidate, each a count of passes divided by NN. The ship rule is blunt: R=0R = 0 or the candidate stays home. A higher P2P_2 never overrules it, because the next section shows how a rising headline hides breakage.

A 40-case Tuesday

Your golden set holds N=40N = 40 cases. The live prompt V1 passes 34 of them:

P1=34/40=0.850=85.0%P_1 = 34 / 40 = 0.850 = 85.0\%

The candidate V2 passes 35:

P2=35/40=0.875=87.5%P_2 = 35 / 40 = 0.875 = 87.5\%

A dashboard that shows only headlines calls this a win and ships it. The buckets say otherwise. V2 fixes F=3F = 3 old failures and regresses R=2R = 2 old passes, so U=40−3−2=35U = 40 - 3 - 2 = 35 cases behave exactly as before. Check the split: 3232 cases pass under both prompts, 33 fail under both, and 32+3=3532 + 3 = 35 unchanged. V2's passes are the 3232 shared ones plus the 33 fixed, which is 3535. V1's were the 3232 shared plus the 22 since regressed, which is 3434. Everything reconciles, and the verdict is still no: R=2R = 2, so V2 stays home despite the prettier headline.

What it costs, and when to skip it

The harness isn't free. You pay for two full runs on every candidate, plus the judge calls if a model scores the outputs, plus the upkeep of the set itself as your traffic shifts. A stale set is the quiet way this dies: the suite stays green while real users ask things it never checks.

So skip it for throwaway work. A one-off analysis prompt with no users and no second run doesn't need a harness. Your eyes are the suite. Use it the moment a prompt serves repeat traffic or another system reads its output. And remember the assumption holding all of this up: the scorer is stable and the set mirrors what users actually send. When the judge version drifts or the set leaks into training data, the counts lie with a straight face. Whole-run scoring for agents, where the steps matter as much as the ending, gets its own treatment in agent evaluation, and retrieval answers with cited chunks in RAG evaluation.

Code

The point of this snippet is the bucketing, not the arithmetic. It classifies every case by its pair of verdicts on the same 40-case set from above, then prints the two headlines and the gate.

def classify(baseline: list[bool], new: list[bool]) -> dict[str, int]:    """Bucket each case by its pair of verdicts."""    fixed = sum(1 for b, n in zip(baseline, new) if not b and n)    regressed = sum(1 for b, n in zip(baseline, new) if b and not n)    unchanged = len(baseline) - fixed - regressed    return {"fixed": fixed, "regressed": regressed, "unchanged": unchanged}

baseline: list[bool] = [True] * 34 + [False] * 6new: list[bool] = [True] * 32 + [False] * 2 + [True] * 3 + [False] * 3
print(classify(baseline, new))  # -> {'fixed': 3, 'regressed': 2, 'unchanged': 35}print(f"{sum(baseline) / len(baseline):.1%}")  # -> 85.0%print(f"{sum(new) / len(new):.1%}")  # -> 87.5%print(classify(baseline, new)["regressed"] > 0)  # -> True

The last line is the gate: True means the candidate stays home. Your CI reads that boolean, not the headline.

Watch Out For

The judge moved, not your prompt

Monday's run shows 2 regressions. Tuesday's rerun of the identical prompt pair shows none, and nobody touched the prompt. You'll burn a day re-reading your diff for a change that never existed.

The cause sits outside your files. The scorer model got a version bump, or it samples with temperature above zero, so the same output passes once and fails next. The fix is pinning: lock the judge model version and run it at temperature 0, rerun baseline and candidate in the same window, and count a regression only if it repeats on a second run.

A golden set that only remembers the wins

The suite glows green for months while users keep re-reporting the same old failure. Each report feels like a new bug, but it's the one you fixed in spring and never wrote down.

The cause is a set built once from easy traffic and never fed. The fix is a habit: every failure that reaches a user becomes a case, expected outputs stay frozen in version control, and someone re-samples recent traffic into the set on a schedule. A suite that can't fail is decoration.

The Quick Version

  • A prompt edit is a deployment: one fixed case can break working ones, so no edit ships on argument.
  • A golden set is a frozen, versioned list of past inputs with demanded answers, rerun against every candidate.
  • Each case lands in exactly one bucket: fixed, regressed, or unchanged.
  • The ship rule is blunt: any regressed case blocks the candidate, even when the headline pass rate rises.
  • In the worked example, V2 moves the headline from 85.0 to 87.5 percent with 3 fixed and 2 regressed, and it stays home.
  • Pin the scorer and refresh the set, or the counts will lie with a straight face.