Tokenization Artifacts
Frequency-built vocabularies file the same word under several spellings, one token ID per surface form. Each ID gets its own embedding, so the model must relearn one meaning several times.
Why Does This Exist?
Your model never reads text. It reads integers, and a tokenizer decides which integer each stretch of characters becomes. Most tokenizers learn that mapping by counting: they merge the byte strings that appear most often in some large corpus and file each merge as a vocabulary entry. Counting is cheap and needs no dictionary. It is also blind to meaning, so the vocabulary it builds is full of near-duplicates and odd entries. Those quirks are tokenization artifacts, and this page is about them.
You feel them as mysteriously uneven behavior. A model nails a word in the middle of a sentence but fumbles the same word at the start of one. It writes fluent paragraphs yet can't tell you how many times a letter appears in a word it just used. A strangely formatted prompt produces confident nonsense. The weights are rarely the culprit. The text was sliced into surprising pieces before the model ever ran, and everything downstream inherited the surprise.
The one idea you need first. A token ID is the integer the model actually reads, and each ID owns one row in the model's embedding table, the lookup that turns IDs into vectors. The slicing itself lives on tokenization, the rows live on embeddings, and the merge algorithm that causes most of this lives on byte-pair encoding. Here you get the damage report: why one meaning ends up with several IDs, what that costs, and the two traps practitioners hit most.
Think of It Like This
One phone number saved under three contacts
You save a number under "Mom", then again under "mom", then a third time under " Mom" with an accidental leading space. Your phone treats those as three different people. If the number changes, you must update all three entries, and the one you rarely open goes stale while the favorite stays fresh. Worse, a pocket dial once saved a wrong number as a contact, and calling it back reaches a stranger.
Token IDs work the same way. Each surface form gets its own entry and its own learned row, frequent forms stay fresh while rare ones go stale, and glitch entries point at noise.
Where it stops: you typed those contacts by hand, but nobody hand-writes a vocabulary. Merges are learned by counting, and a string the vocabulary never filed does not fail to save. It shatters into smaller pieces, letters if necessary, which the model then has to reassemble from parts that never carried the whole meaning.
How It Actually Works
Call the tokenizer's piece list , its vocabulary. For any string , write for the integer the model actually reads. The model holds an embedding table with one row per ID; write for the row belonging to ID , and for the embedding width, the count of numbers stored in each row. So the price of one vocabulary entry is exactly numbers, and the price of a duplicated meaning is numbers times the count of its variants.
Why one meaning gets several IDs
A space is an ordinary byte to the tokenizer, often displayed as Ġ, and case changes the bytes too. So with a leading space, , and are three different byte strings that travel three different merge paths and land as three different entries in , with three different IDs. Nothing in the merge step knows they mean the same thing, because the merge step only ever counted.
Training then treats the three IDs as strangers. Gradient updates land only on rows the training text actually shows, so the frequent variant's row sharpens while the rare variant's row stays weak. Same meaning, unequal knowledge. A related number is fertility, the count of tokens one word costs: a familiar surface form costs one token while an oddly spaced or capitalized twin can cost several, and since context windows are priced in tokens, the same sentence can cost more simply because of how it was typed.
A worked toy example
Take a toy vocabulary with three surface forms of one word and an embedding width :
The table holds one row per ID, so numbers. One meaning needs a single row of numbers, so numbers in this toy are pure redundancy: three rows to learn, store, and keep consistent instead of one. Now suppose the toy corpus shows ID 3 a hundred times and ID 9 twice. Nearly every update sharpens and almost none touch , which is exactly how a real model ends up fluent in a word mid-sentence and shaky on its sentence-start twin. The diagram above draws this fixture: three forms, three IDs, three four-cell rows, twelve cells where four would do.
Glitch tokens and shattered words
Two edge cases complete the picture. A glitch token is a vocabulary entry the tokenizer filed because some quirk repeated often in its training scrape, yet the model's own training barely shows. Its row receives almost no gradient signal and sits near its starting values, so forcing the model to read or write that ID produces erratic output. The fix is dull on purpose: treat unknown IDs as corrupt input and refuse or normalize them instead of reasoning with them.
A shattered word is the mirror image. Pad a familiar word with odd spaces or case and the vocabulary has no entry for the result, so it falls back to small pieces, single letters in the extreme. Letter rows carry almost none of the whole word's meaning, and the model must reassemble the sense from crumbs. That's why spelling, rhyming, and letter-counting tasks feel absurdly hard for fluent models: you asked about letters, but the model was handed whole chunks.
Name the trade-off plainly. Frequency merging buys short sequences, fewer tokens per sentence, at the cost of units that slice meaning awkwardly. It assumes a frequent string is a meaningful unit, and that assumption fails for names, code, morphology, and scripts the merge corpus barely covered. Do not fix uneven behavior by filing a whole-word entry for every variant you meet. The vocabulary is finite, every entry costs numbers plus the training sightings needed to sharpen them, and each patch teaches the model one more spelling instead of the underlying word.
Code
The snippet below replays the toy fixture as a lookup, so you can see the duplication as data rather than prose. Three surface forms, three IDs, and a table holding twelve numbers for one meaning.
vocab: dict[str, int] = { ' Apple': 7, # leading space: its own merge path 'Apple': 3, # the frequent form ' apple': 9, # lowercase with space: rarest of the three}
def encode(form: str) -> int: return vocab[form] # toy lookup: each surface form owns one row
D: int = 4 # embedding width: numbers stored per token IDtable: list[list[float]] = [[0.0] * D for _ in range(len(vocab))]
forms = [' Apple', 'Apple', ' apple']print([encode(f) for f in forms]) # -> [7, 3, 9]print(len(table) * D, "numbers stored") # -> 12 numbers storedprint(1 * D, "numbers needed for one meaning") # -> 4 numbers needed for one meaningTwelve numbers stored against four needed, and the gap is the artifact: every near-duplicate row in a real table repeats this pattern, each one demanding its own sightings before its meaning sharpens.
Watch Out For
Asking the model to count letters it never saw
The symptom is specific and humbling: the model uses a word correctly, then tells you with full confidence that it contains the wrong number of a letter. Nothing is wrong with its knowledge. It read one integer for the whole chunk, so the letters were never in its input the way they are in yours.
Ask it to spell the word out first, one letter per line, and then count over the spelled letters. For anything that matters, skip the model and count in code. Letter questions go to string functions, never to next-token prediction.
Grading a model on its worst-spelled variant
The symptom is uneven scores that look like a capability gap: the model answers cleanly with one phrasing and stumbles when the prompt changes only in casing or spacing. Before you conclude it cannot do the task, print the tokenizer output for the exact string you sent. You will often find the hard variant shattered into far more pieces than the easy one.
Fix the test before you blame the weights. Normalize casing and spacing across evaluation prompts so every variant gets the same segmentation, and keep one debugging habit: the exact string's token IDs on screen before any theory about the failure.
The Quick Version
- A tokenization artifact is a vocabulary entry that exists because a byte string was frequent, not because it is a meaningful unit.
- A leading space or a capital letter makes a different byte string, so one word can own several token IDs with one embedding row each.
- Rare variants get few training updates, so their rows stay weak while the frequent twin looks fluent.
- Glitch tokens are filed but almost never trained, so forcing them produces erratic output that no prompt trick can rescue.
- Models read whole chunks rather than letters, so counting, spelling, and rhyming questions need spelled-out letters or plain code.
- Frequency merging trades shorter sequences for awkward units, and patching the vocabulary variant by variant never pays off.