Memorization and Extraction
Your model memorized a training passage word for word, and a prompt holding its first half can pull out the second half. Scrub secrets before training, because no filter can reliably catch them after.
Why Does This Exist?
A language model trains on text scraped from the open web: documentation, forums, repos, books. That scrape sweeps up things nobody meant to teach it. A key pasted into a public config. A private address quoted in a bug report. One license header stamped on ten thousand files.
Most of that text teaches general patterns. Some of it gets stored instead. A model with enough capacity files away exact strings, and from the outside you can't see which ones. Then anyone who guesses the first half of a stored string can prompt the model with it and read off the second half. That's the whole threat: training data goes in, and a prefix prompt pulls pieces of it back out.
This page covers why storage happens, how a prefix turns into a leak, how to measure the risk with planted canaries, and what actually stops it. The general mechanism behind the storage lives in overfitting. Other input-side attacks that pull unwanted behavior out of models live in jailbreaking and prompt injection.
The one idea you need first: a model writes by next-token prediction. It reads the tokens so far and bets on the next one, over and over. Everything on this page follows from that. Training rewards the bets that match the corpus, and extraction just feeds the model a prefix so its best bet is the memorized continuation.
Think of It Like This
A song you never meant to learn
You rode the same bus route for a year, and the driver always played the same station. You never tried to learn any of it. Then a friend hums the first bar of one jingle, and the rest plays in your head without your permission, every word in order.
That is memorization: repetition burned a string into storage without study. The hum is the prefix prompt. It works because recall is associative. Supply the start, get the rest.
Where it stops: you know the jingle isn't yours, and you can choose not to sing it aloud. The model has neither check. It carries no tag saying a string came from one private document, and at low temperature it can't hold back its most likely continuation.
How It Actually Works
Why training stores exact strings
Training minimizes the negative log likelihood of the corpus: at each position the model pays , where is its predicted probability, is the true next token, and is every token before it. In plain words: surprise at the real continuation, averaged over everything it reads.
A string seen once nudges the weights once. A string seen a thousand times nudges them a thousand times, carving a groove so that whenever its prefix appears, the stored continuation becomes the top bet. Duplication is the dominant cause, which is why deduplicating the corpus helps most.
Capacity is the second cause. A bigger model has more room to file strings away while still learning general patterns, so frequency is no safety guarantee. Rare strings can stick too. Assume any secret in the corpus is retrievable and you won't go far wrong.
How a prefix turns into a leak
Extraction needs no cleverness once storage has happened. The attacker resupplies the opening tokens of a stored string, sets the sampling temperature to 0 so the model always picks the top bet, and reads the completion. A direct question ("what is the password?") trips refusal training. A bare prefix looks like an ordinary continuation task, so it usually doesn't.
The nastiest version needs no known secret at all. Repeating a single token hundreds of times can knock the model off its learned style, and when it falls off, what surfaces is sometimes raw memorized text from unrelated documents. The prefix does the aiming. This derailing fires blindly and reads whatever comes up.
Worked example: exposure in bits
Practitioners plant a canary: a random secret inserted into the training data, then ranked against every other string the model could have produced. Call the candidate count (how many possible suffixes exist) and the canary's rank (1 means the model finds the true suffix most likely). Its exposure is:
reads as how many bits of uncertainty the model removed. Say the suffix space holds possibilities (, so ) and the model ranks the true suffix nd ():
Nine of ten bits gone: the model has nearly identified the secret, and the prefix prompt in the diagram recovers it. Contrast a canary ranked th (): bit, barely better than guessing. Same corpus size, same model, wildly different risk. That gap is what deduplication buys you: fewer copies, worse rank, lower exposure.
The trade-off is blunt. Scrubbing and deduplicating cost data, compute, and coverage: every filter drops some legitimate text, and secret-scanning adds pipeline work before each run. What it buys is the only reliable protection, because post-training fixes rest on a broken assumption. An output filter assumes a secret looks distinctive enough to catch, but a filter staring at a plausible key can't tell a stolen one from an invented one. And memorization itself isn't the enemy: you want the model to keep facts, idioms, and standard code patterns. The line sits between learning a fact and storing a string, and only curation before pretraining walks it.
Code
The snippet below is the measurement harness teams run on canaries: convert a planted secret's rank to exposure bits, then score how many secrets come back character for character.
import math
def exposure_bits(candidate_space: int, rank: int) -> float: """Exposure of a planted canary, in bits.""" return math.log2(candidate_space) - math.log2(rank)
def exact_match_rate(secrets: list[str], recovered: list[str]) -> float: """Share of secrets recovered character for character.""" hits = sum(1 for s, r in zip(secrets, recovered) if s == r) return hits / len(secrets)
print(exposure_bits(1024, 2)) # -> 9.0print(exposure_bits(1024, 512)) # -> 1.0
secrets = ["Q7-29-BLUE", "Z3-11-GREEN", "K9-44-RED", "M2-78-BLACK"]recovered = ["Q7-29-BLUE", "Z3-11-GREEN", "K9-44-RED", "something else"]print(exact_match_rate(secrets, recovered)) # -> 0.75The recovered list stands in for real model outputs, which you collect by prompting with each canary's prefix. Run this before every release: if exposure climbs or the match rate leaves zero, something you added is being stored rather than learned.
Watch Out For
Filtering outputs instead of cleaning inputs
Symptom: your guardrail blocks some completions and waves others through, yet secrets still escape in paraphrase or in pieces across turns. It can't do better. A filter seeing a realistic key has no way to know whether the model memorized it from your corpus or invented it for a tutorial, so it alternates between false alarms and misses.
Fix: move the work before training. Deduplicate the corpus, scan it for secrets and personal data, and canary-test what remains. Keep output filters, but treat them as a backstop for accidents, never as the defense.
Counting copies and calling rare secrets safe
Symptom: a secret appears once, buried in boilerplate, so the team signs off on it. Then an attacker pastes the boilerplate as a prefix and reads the secret out, because capacity lets infrequent strings stick and the attacker supplies the half the model needs.
Fix: never clear a secret by rarity. If it's in the corpus, assume it's retrievable, canary-test it like everything else, and delete what fails. Frequency sets the odds, not the possibility.
The Quick Version
- A model stores exact training strings when repeated updates turn a prefix into its top bet, and duplication is the main driver.
- A prefix prompt recovers a stored string: resupply the opening tokens and the best continuation is the memorized rest.
- Exposure measures the risk in bits, and the worked canary scores 9 bits against 1 for a well-buried one.
- Deduplicate and scrub secrets before training, because output filters can't tell a stolen key from an invented one.
- Assume every secret in the corpus is retrievable, and canary-test each release to prove otherwise.
- Memorization isn't all bad: facts and idioms should stick, while private strings must never reach the corpus.