Tokenization in NLP

Tokenization cuts raw text into pieces models can index. Frequent words stay whole while rare ones split, so one word can become three tokens.

The word unbelievably splits into un plus believ plus ably with IDs 4827, 23419 and 9421, so three tokens replace twelve characters.

Why Does This Exist?

Models only do math on numbers, so raw strings must become discrete units first. Pick the wrong unit and everything downstream pays. Whole words keep meaning yet explode the vocabulary, since every form, typo and name wants its own row. Single characters keep the table tiny yet stretch one word into a dozen steps and force the model to relearn spelling before meaning.

Tokenization picks the middle path. Frequent words stay whole, rare words split into familiar chunks, and anything unseen still breaks into known pieces. You get a bounded table with no dead ends.

This page covers the split and the ID lookup. Cleaning before the split lives in text normalization, vectors after the lookup in word embeddings, and the full order in the NLP pipeline.

The one idea you need first

A vocabulary is a lookup table from piece to ID. The tokenizer cuts text, then each piece becomes one integer the model embeds. If a piece has no row, the model has no vector, which is why coverage matters more than cleverness.

Think of It Like This

A box of reusable word tiles

You own a tile game with whole words, common chunks and single letters. A friend asks for an odd word you never stocked. You do not carve a new tile on the spot. You build it from tiles you have: un plus believ plus ably.

Tokenization plays the same game. Whole common words get one tile each. Rare words get built from frequent chunks. Single letters cover the rest, so every string is buildable from a fixed box.

Where it stops: tiles hide spelling inside the build. Once unbelievably becomes three IDs, the model sees the IDs, not the letters. Counting letters from IDs is guesswork, which is why such models stumble on spelling games.

How It Actually Works

Word splits keep meaning and explode the table

Split on spaces and punctuation. The sentence "I love NLP" becomes ["I", "love", "NLP"]. Tokens match what you read, so the start feels natural.

The cost is coverage. Call the table size $V$ . Here $V$ is the number of rows you store. English needs hundreds of thousands of rows for full word coverage, and each new typo or name still misses. Misses collapse to one unknown token, which deletes the most telling word in the sentence.

Character splits cover everything and stretch time

Treat each letter as a token. The word "NLP" becomes ["N", "L", "P"]. The table shrinks to an alphabet plus marks, and nothing is ever unknown.

The cost is length. The word unbelievably has 12 letters, so one word costs 12 steps. Transformer work grows fast with steps, so you pay in compute what you saved in rows. Meaning also thins, since single letters carry almost none.

Worked subword split

Subword learns frequent chunks from a corpus and reuses them. Common words stay whole. Rare words split. The marker ## means continuation of the same word in WordPiece style.

The word unbelievably splits into ["un", "##believ", "##ably"]. Here "un" is a frequent prefix, "##believ" is the root continuation, and "##ably" is the suffix continuation. In this fixture the vocab maps them to IDs $[4827, 23419, 9421]$ . One word, three tokens, three vectors.

Twelve characters become three tokens. The table stays near tens of thousands instead of hundreds of thousands, and no input is ever unbuildable. Those are the numbers in the diagram above.

The trade is plain. Larger tables keep more whole words and shorten runs, at the price of memory and rare rows that train poorly. Smaller tables cover everything with chunks, at the price of longer runs that hide spelling.

Code

The worked split as a lookup, so the one to three to three counts can be checked.

1def tokenize(word: str) -> list[str]:2    pieces: list[str] = ["un", "##believ", "##ably"]  # fixture split for unbelievably3    return pieces4
5
6def to_ids(pieces: list[str], vocab: dict[str, int]) -> list[int]:7    return [vocab[p] for p in pieces]  # each piece owns one row8
9
10vocab: dict[str, int] = {"un": 4827, "##believ": 23419, "##ably": 9421}11pieces: list[str] = tokenize("unbelievably")12print(pieces)  # -> ['un', '##believ', '##ably']13print(to_ids(pieces, vocab))  # -> [4827, 23419, 9421]14print(len("unbelievably"))  # -> 12

Twelve characters become three pieces and three IDs. The lookup never misses because chunks cover the tail.

Watch Out For

Unknown tokens that delete the key word

Symptom: a word level table meets a new drug name or handle, emits one unknown token, and the classifier loses the exact signal it needed. Accuracy looks fine on common text and collapses on the rare slice.

Fix: use subword tables so rare forms split into known chunks. Keep an eye on the unknown rate per slice, not just overall, before you trust the vocab.

Splits that hide spelling and counts

Symptom: you ask how many r letters sit in strawberry, the model sees two opaque IDs instead of letters, and it miscounts with full confidence. The split threw away the view your question needed.

Fix: keep raw strings beside IDs for spelling checks, or add a character view for such tasks. Never ask a subword model to count letters from IDs alone.

The Quick Version

Tokenization cuts raw text into pieces a model can index as IDs.
Word splits keep meaning yet explode the table and still miss new forms.
Character splits cover everything yet stretch one word into many slow steps.
Subword keeps frequent words whole and builds rare ones from chunks.
Worked case: unbelievably becomes un plus believ plus ably with IDs 4827, 23419 and 9421.
Twelve characters cost three tokens, so bounded tables stay buildable without hiding spelling workarounds.