Tokenization in NLP
Tokenization breaks raw text into smaller, manageable units (tokens) — like words, subwords, or characters — so that models can process discrete elements instead of an unbroken character stream.
Why Does This Exist?
Machine learning models, from the simplest logistic regression classifiers to the most complex transformer networks, fundamentally require numerical input. They cannot directly process a string of text like "I love natural language processing." Before any linguistic analysis or embedding can take place, we must first establish a discrete unit of text that the model will learn to understand. This process of breaking raw, continuous text into these discrete, manageable units is called tokenization.
If we choose the wrong atomic unit, the entire downstream system suffers. If we decide to use entire words as our fundamental units (word-level tokenization), our vocabulary becomes impossibly large. We would need a separate representation for every possible word, every conjugated form, every pluralization, and every common misspelling. English alone has hundreds of thousands of words, and many other languages are even more complex.
Conversely, if we decide to use single characters as our fundamental units (character-level tokenization), the vocabulary shrinks to a highly manageable size (just the alphabet, digits, and punctuation). However, characters by themselves carry no semantic weight. The model would be forced to spend a significant portion of its capacity just learning how characters assemble into words before it could even begin to learn what those words mean. Tokenization exists to find the optimal sweet spot between these two extremes, producing units that are small enough to keep the vocabulary size practical, but large enough to retain intrinsic semantic meaning.
Think of It Like This
Breaking down a chemical compound
Imagine you are trying to analyze a complex chemical compound to understand its properties. You wouldn't attempt to analyze the entire block of matter as a single indivisible object—that would tell you nothing about its structure. On the other hand, you wouldn't break it all the way down into individual protons, neutrons, and electrons, because at that level, the unique chemical properties of the substance are completely lost.
Instead, you break the compound down into molecules, and then into individual atoms. Atoms are the fundamental building blocks—they are small enough to be a manageable, finite set of elements (like the periodic table), but large enough to dictate the properties of the molecules they form. Tokenization is exactly the same process for language: it is the act of finding the linguistic "atoms" that are small enough to be manageable, but large enough to retain their meaning.
How It Actually Works
The evolution of tokenization has generally moved from simpler rule-based approaches to sophisticated statistical methods that learn the optimal splits from data.
1. Word-level Tokenization
The earliest and conceptually simplest approach is to split text by whitespace and punctuation marks.
- The sentence
"I love NLP!"becomes the sequence["I", "love", "NLP", "!"]. - This approach makes intuitive sense to humans because tokens correspond directly to recognizable words.
- However, it suffers from a massive vocabulary size and the Out-of-Vocabulary (OOV) problem. When a model encounters a word it has never seen during training (a new slang term, a typo, or a rare name), it has no way to represent it, typically substituting an uninformative
<UNK>(unknown) token.
2. Character-level Tokenization
To eliminate the OOV problem entirely, one can treat every individual character as a token.
- The word
"NLP"becomes["N", "L", "P"]. - This approach has a tiny vocabulary size, meaning the model's embedding matrix takes up very little memory. It can theoretically represent any text, no matter how unusual.
- The severe downside is that sequences become extremely long. A sentence of 20 words might turn into a sequence of 100 characters. For models like transformers whose computation scales quadratically with sequence length, this is computationally prohibitive.
3. Subword Tokenization (The Modern Standard)
Modern Natural Language Processing systems rely almost exclusively on subword tokenization. This approach learns the most frequent character combinations in a training corpus. Common words remain intact as single tokens, while rare words are split into recognizable subword chunks.
- The rare word
"unbelievably"might be split into["un", "##believ", "##ably"](using the WordPiece algorithm). The##indicates that a token is a continuation of a previous word rather than the start of a new one. - Byte-Pair Encoding (BPE): Starts with a vocabulary of individual characters. It iteratively counts the most frequent adjacent pairs of tokens and merges them into a new, single token. It repeats this merging process until a predefined vocabulary size is reached.
- WordPiece: Very similar to BPE in its iterative merging, but instead of merging the most frequent pairs, it merges the pair that maximizes the likelihood of the training data.
- SentencePiece: Unlike BPE and WordPiece which often assume text is pre-tokenized into words (usually by spaces), SentencePiece treats whitespace as just another character (often represented as
_). This allows it to learn tokenization without any language-specific pre-segmentation, which is crucial for languages like Chinese or Japanese that do not use spaces between words.
Code
The snippet below demonstrates how a simple Byte-Pair Encoding (BPE) algorithm iteratively merges the most frequent adjacent characters. This is a simplified version of what runs behind the scenes when training a modern tokenizer.
import collectionsfrom typing import Dict, Tuple
def get_stats(vocab: Dict[str, int]) -> Dict[Tuple[str, str], int]: pairs = collections.defaultdict(int) for word, freq in vocab.items(): symbols = word.split() for i in range(len(symbols) - 1): pairs[symbols[i], symbols[i + 1]] += freq return pairs
def merge_vocab(pair: Tuple[str, str], v_in: Dict[str, int]) -> Dict[str, int]: v_out = {} bigram = " ".join(pair) replacement = "".join(pair) for word_in in v_in: # Replace the space-separated bigram with the merged version word_out = word_in.replace(bigram, replacement) v_out[word_out] = v_in[word_in] return v_out
vocab = { "l o w </w>": 5, "l o w e r </w>": 2, "n e w e s t </w>": 6, "w i d e s t </w>": 3}
num_merges = 3for i in range(num_merges): pairs = get_stats(vocab) if not pairs: break best = max(pairs, key=pairs.get) vocab = merge_vocab(best, vocab) print(f"Merge {i + 1}: {best} -> {''.join(best)}")# -> Merge 1: ('e', 's') -> es# -> Merge 2: ('es', 't') -> est# -> Merge 3: ('est', '</w>') -> est</w>This toy example shows how the frequent sequence e, s, t gets merged into the subword est</w>, which will now be treated as a single token by the model.
Watch Out For
The Out-of-Vocabulary (OOV) problem
If your word-level tokenizer encounters a word it wasn't explicitly trained on, it usually replaces it with an <UNK> (unknown) token. This destroys critical information. If a medical NLP system replaces a rare drug name with <UNK>, the model loses the most important piece of data in the sentence. This is exactly why modern models universally use subword tokenization, which can break any unknown word down into known subword or character tokens, ensuring no text is ever completely unrepresentable.
The Quick Version
- Tokenization is the mandatory first step in any NLP pipeline, breaking continuous text into discrete, numerical units.
- Word-level tokenization creates massive, unwieldy vocabularies and struggles heavily with unknown words or misspellings.
- Character-level tokenization solves the vocabulary size problem but creates excessively long sequences that are computationally expensive to process.
- Subword tokenization algorithms (like BPE, WordPiece, and SentencePiece) are the modern standard, dynamically balancing vocabulary size and semantic density by breaking rare words into common chunks.