Red Teaming
You attack your own model on purpose before strangers do. Thousands of hostile prompts go in, the breaks get patched, and launch happens after the ambush, not before.
Why Does This Exist?
Friendly testing asks whether the model works when asked nicely. Attackers ask what it does when asked hostilely. A model can pass every demo prompt and still hand disallowed help to the first stranger who reframes the request, because safety training teaches refusal without erasing the underlying knowledge.
You cannot fix breaks you have not found, and strangers will find them fast once you launch. So you stage the ambush yourself first. A red team is a group told to break your model on purpose, weeks before release, while the fixes are still cheap and quiet.
This page covers the drill itself: who attacks, what they throw, how you score it, and what changes afterward. The attacks they throw are taught in jailbreaking. The screens you build from the findings are guardrails. Grading the results is safety evaluation.
You need one idea first. Refusal is a habit laid over knowledge, not a wall around it. Pretraining has seen the shape of disallowed answers, and alignment added a style that says no. Red teaming measures how easily a clever prompt peels that style off.
Think of It Like This
A fire drill in a new school
A new school opens next month. Before any child arrives, the principal pulls the alarm on a Tuesday morning with only the staff inside. Most doors swing open and the halls empty on time. One stairwell door sticks. Nobody is hurt, nobody panics, and the caretaker fixes the hinge that afternoon. Then they drill again.
The mapping is direct. The staff are your testers, the alarm is the attack prompt, the doors that opened are the 9,999 refusals, and the stuck door is the 1 jailbreak. The fixed hinge is the new guardrail rule, and the second drill is the retest.
Where it stops: a drill only tests the exits you thought to try, with a calm crowd. A real fire brings smoke and panic you never rehearsed. Same with models. A clean drill means your list ran out, not that no attack exists.
How It Actually Works
Call the total probes , so is every adversarial prompt the team fires. Call the successes , so is the prompts that got a disallowed answer out. Call the blocked count , so . The scoreboard is one ratio, the attack success rate:
A low ASR is good. It says most attacks bounced off. But one success is enough to matter, because attackers only need one working phrasing while you must block them all.
Pick the targets first
A red team never starts by typing. It starts by listing what would hurt for this deployment. A medical helper must never output a lethal dose. A support bot must never promise a refund it cannot give or insult the customer. An agent with tools must never let a prompt talk it into deleting a database. Each harm on the list becomes a goal the team aims at, and anything off the list is out of scope for this round.
Throw brains first, then volume
Humans go first because people invent what scripts cannot. A tester tries roleplay, encoded words, translations, and long buried instructions, hunting for one novel break. Each success teaches the team which technique works, and the full catalog of those tricks is jailbreaking.
Then machines take over for breadth. An attacker model generates thousands of variations on the working tricks, probing every corner the humans found. Humans find the first hole. Automation checks whether that hole has a thousand siblings, including cousins like indirect prompt injection hiding in tool output.
Turn each break into a regression case
A success is not a bug report to file away. It becomes permanent ammunition. The team records the exact prompt, extracts the technique behind it, and hands three things to the builders: a new guardrail rule that blocks the technique, a fresh eval item so the break is retested every release, and sometimes a training example so alignment learns it directly. Then the drill runs again to confirm the fix held.
Worked example: 10,000 probes, 1 break
Say the round fires prompts. The target refuses and one roleplay prompt slips through, so .
The block rate is , which sounds like victory until you read it the attacker's way. They tried ten thousand phrasings and now own one that works. The team generalizes that prompt into its technique, writes one guardrail rule against the technique, adds the whole paraphrase family to the eval set, and reruns. The diagram above draws this same round: 10,000 in, 9,999 stopped, 1 break feeding the patch.
The trade sits in the open. Manual testers buy depth at high cost per probe. Automation buys breadth at low cost per probe but only replays tricks it already knows. Skimp on either and you miss your kind of break.
Skip the full drill when there is nothing to steal and nobody strange can type. A local demo with no tools, no secrets, and no strangers is fine with refusal plus a quick pass. Once strangers, tools, or sensitive data enter the picture, run the drill.
Code
The point here is the scoreboard, not the attacks. Leadership tracks this one ratio across rounds: did the latest patch round push the success rate down without breaking innocent prompts.
ATTEMPTS = 10_000SUCCESSES = 1
def attack_success_rate(successes: int, attempts: int) -> float: return successes / attempts
def blocked_rate(successes: int, attempts: int) -> float: return 1.0 - successes / attempts
asr = attack_success_rate(SUCCESSES, ATTEMPTS)print(f"{asr:.4%}") # -> 0.0100%print(round(blocked_rate(SUCCESSES, ATTEMPTS), 4)) # -> 0.9999One success in ten thousand is an ASR of 0.0100% and a block rate of 0.9999. Same pair as the worked example and the diagram. Watch both numbers across rounds: ASR falling means the patches hold, while a rising false-refusal rate on innocent prompts means the patches are biting too deep.
Watch Out For
Patching the prompt instead of the technique
Symptom: you block the exact jailbreak string, the metric goes green, and next week a paraphrase of the same trick walks straight through. Your ASR looked fixed because the eval only retested the string you already knew.
Fix: generalize before you celebrate. Name the technique (roleplay framing, say), write the guardrail rule against that technique, and add a paraphrase family you did not train on to the eval set. Retest with fresh phrasings from a different tester. A rule that only matches yesterday's string is a speed bump.
Reading zero finds as proof of safety
Symptom: a round ends with no successes, the team ships with confidence, and the public breaks the model within days. The clean report measured the list, not the model.
Fix: treat a clean round as "our list ran out." Rotate in new testers, seed fresh attacks from the latest public jailbreak tricks, widen the harm list, and keep the drill running every release. Never ship on a single clean round. Any model falls to enough tries; the drill raises the cost, it never certifies immunity.
The Quick Version
- Red teaming attacks your own model with hostile prompts before launch so strangers cannot be the first to break it.
- Humans probe first for novel breaks, then an attacker model replays the working tricks at volume to find every sibling.
- The scoreboard is the attack success rate, successes divided by total probes, and one success outweighs thousands of blocks.
- In this page's drill 10,000 probes yield 1 jailbreak, and that single break becomes a guardrail rule plus a permanent eval case.
- A clean round means the attack list ran out, never that the model is safe, so the drill repeats every release.