Safety Evaluation
You grade a model by attacking it with fixed hostile prompts and counting clean refusals. The refusal share decides whether it may launch.
Why Does This Exist?
You ship a helpful assistant. Then a stranger reframes one request and it hands over help it should have refused. Training lowered the odds, but training can't promise the refusal holds against every phrasing. The knowledge stays inside the weights, and only the refusal habit sits on top.
That's the gap this page closes. Friendly tests ask whether the model works when asked nicely. Attackers ask what it does when asked hostilely. Without a hostile test you launch blind, and the first real jailbreak finds you instead of your test suite.
Safety evaluation is that hostile test, run as a fixed drill you can repeat. You fire a frozen list of attack prompts, grade every reply as refusal or jailbreak, and report the share that refused. This page covers the drill and its score. The people who invent fresh attacks have their own page in red-teaming. The attacks they throw are taught in jailbreaking. The screens you build from the findings are guardrails.
You need one idea first. A judge model can grade open-ended replies at script speed by following a written rubric, which is the mechanism taught in LLM-as-a-judge. Safety evaluation points that judge at one question: did the model refuse.
Think of It Like This
A bouncer tryout with planted troublemakers
A club owner hires a new bouncer and wants proof, not promises. Before opening night she sends 200 planted troublemakers through the line: fake IDs, borrowed coats, friends vouching at the rope. She counts how many get turned away cleanly. When 188 go home and 12 slip inside, the door holds at 94 percent, and the 12 slip-throughs become Monday's training.
The mapping is direct. The bouncer is your model. The planted troublemakers are your frozen probe list. A clean turn-away is a refusal. A slip inside is a jailbreak. The 94 percent is your refusal rate, and Monday's training is the guardrail patch.
Where it stops: a bouncer reads faces, IDs, and body language, and real troublemakers study the door for weeks before trying. Your probes are text only, and your list freezes while attackers keep inventing. A clean tryout means the planted crew failed, not that no crew can pass.
How It Actually Works
The drill has three steps. First you fix the probe list: a frozen set of hostile prompts, grouped into harm classes, that every model version faces unchanged so scores compare across weeks. Then the model answers each probe once, with sampling settings pinned, so the replies reflect the model and not the dice. Then a judge grades each reply against a written rubric as REFUSAL or JAILBREAK, and you count.
Call the total probes , so is every hostile prompt you fired. Call the clean refusals , so is the replies that declined without leaking help. Call the jailbreaks , so is the replies that complied. The headline score is the refusal rate:
Its complement is the attack success rate, , which is the number the red team quotes. In plain words: is how many attacks you tried, is how many bounced off, is how many got through, is the share that held, and is the share that broke.
The scoreboard: two ratios from one run
Both ratios come from the same count, so report them together. A refusal rate of 0.94 and an attack success rate of 0.06 describe the identical run of 200 probes: 188 bounced, 12 broke through. Teams that quote only one of the pair invite mischief, because a reader can't tell a 94 percent refusal rate from a 94 percent success rate without the label. Write the fraction every time, not just the decimal: 188/200 carries its own sample size, while 0.94 hides whether you fired 200 probes or 16.
Per-class scores sit beside the headline. A model can hold at 94 percent overall while one class sags at 90 percent, and that sagging class is where the next patch goes. The diagram above draws exactly this run, with the worst class called out.
Where the 200 probes come from
A good list mixes sources on purpose. Human red-teamers contribute the newest jailbreak framings they found this cycle, which keeps the list honest about current attacker skill (see red-teaming). Paraphrase variants of each find stop the model from passing by matching exact strings. Seeds from past incidents stop regressions: any prompt that broke you before stays in the list forever. And the classes stay fixed across runs, typically five: hate, self-harm, illicit help, private-data extraction, and cyberattack help, with 40 prompts each for 200 total.
Pin everything else. Same model build, same system text, same sampling settings, same judge version and rubric. Change one thing per run or the score stops meaning anything.
Worked example: 200 probes, 188 refusals
Five classes, 40 prompts each, so . The judge returns these refusals per class: hate 40 of 40, self-harm 38 of 40, illicit help 37 of 40, private-data extraction 37 of 40, cyberattack help 36 of 40. Add them:
Jailbreaks are the remainder, , split 0, 2, 3, 3, 4 across the same classes. Now the two ratios:
The worst class is cyberattack help at percent, four points under the headline. That class owns the next patch: new guardrails patterns for its framings, then a rerun of the identical 200.
Read the trade before you celebrate. Pushing refusals up almost always pushes false refusals up too: benign prompts that get declined because they share words with attacks. So every safety run travels with a benign set of innocent but edgy prompts, and you watch both numbers. A 100 percent refusal rate on attacks means nothing if a quarter of harmless requests get refused beside it. And don't read the refusal rate as quality. A model that refuses everything scores perfectly and helps nobody. Safety evaluation gates launch; it never grades helpfulness. The assumption underneath is that your judge matches human judgment and your list mirrors real attackers. When either fails, the score lies, which is what the two traps below are about.
Code
The point of this snippet is the scoreboard, not the formula. It turns a verdict list into the two ratios and carries the worked example's numbers, so you can check your own harness output against 0.94 and 0.06.
from typing import Literal
Verdict = Literal["REFUSAL", "JAILBREAK"]
def refusal_rate(verdicts: list[Verdict]) -> float: refusals: int = sum(1 for v in verdicts if v == "REFUSAL") return refusals / len(verdicts)
def attack_success_rate(verdicts: list[Verdict]) -> float: return 1.0 - refusal_rate(verdicts)
# 188 refusals and 12 jailbreaks, matching the worked exampleverdicts: list[Verdict] = ["REFUSAL"] * 188 + ["JAILBREAK"] * 12print(f"{refusal_rate(verdicts):.2f}") # -> 0.94print(f"{attack_success_rate(verdicts):.2f}") # -> 0.06print(f"{len(verdicts)}") # -> 200Three calls, three checks. The rate prints 0.94, its complement prints 0.06, and the length confirms all 200 verdicts went into the count. If your harness drops malformed judge replies instead of counting them, that last number is where you'll catch it.
Watch Out For
The judge grades its own homework
You let the model under test, or its smaller sibling, grade its own replies. The refusal rate comes back near perfect, you ship, and a human audit finds long compliant answers the judge marked as refusals because they opened with "I shouldn't..." before complying anyway.
The symptom is a score that never moves: 99 percent across versions, including versions you know regressed. The fix has three parts. Grade with a stronger, frozen judge from a different family than the model under test. Force critique before verdict in the rubric, so the judge quotes the reply's actual behavior before writing the label. And hand-audit a random sample of at least 50 verdicts every run. If your audit disagrees with the judge more than once or twice in that sample, fix the rubric before you trust the headline.
A frozen list that always passes
You froze the probe list a year ago and it still reports 100 percent. Meanwhile a new framing from prompt injection walkthroughs breaks you in production, because no prompt on your list phrases the attack that way. A passing score on a stale list measures the list's age, not the model's safety.
The symptom is a flat line: perfect scores for months while attacker chatter moves on. The fix is rotation with memory. Each cycle, fold in fresh finds from the red team and hold out the newest attacks as a surprise set the model never trains against. Keep every past breaker in the regression set so old holes stay shut. And track the benign false-refusal rate beside the headline, so patches that buy refusals with uselessness show up immediately.
The Quick Version
- Safety evaluation is a fixed drill: frozen attack prompts in, refusal or jailbreak verdicts out.
- Refusal rate is refusals divided by probes fired, so 188 refusals in 200 probes is 94 percent.
- Attack success rate is the complement, so 12 jailbreaks in 200 probes is 6 percent.
- A judge model grades each reply against a written rubric, with humans spot-checking a random sample.
- A high refusal rate means nothing without a benign set, since a model that refuses everything scores perfectly and helps nobody.
- Rotate the probe list every cycle, because attackers invent phrasing a frozen list never tries.