Generative AI
Tokenization (BPE)
Type any text and watch it get broken into subword tokens live, showing merge rules being applied step by step.
How models see text as statistically common subword chunks formed by merging frequent character pairs.
Stage 1 of 2: Characters
Current text has 7 tokens. Next merge rule applies to "in".
- Token
Language models start by splitting text into characters, the smallest atomic units.
Check your understanding
1 questions in the bank. Each attempt draws a fresh set in a fresh order, so a second go is a real second go.
Byte Pair Encoding (BPE) is the algorithm that most modern large language models use to see the text we type. Instead of breaking text into characters (which is too fine-grained and produces huge context lengths) or into words (which fails on typos, punctuation, and new words), BPE finds the perfect middle ground.
By starting with characters and iteratively merging the most frequently adjacent pairs in the training corpus, the tokenizer builds a vocabulary of subwords. Common words might be a single token, while rare ones are split into pieces.
This lab lets you watch the inference step: applying those learned merge rules to new text, step by step.
Reference
- Token
- A chunk of text (character, subword, or word) assigned a unique integer ID.
- BPE
- Byte Pair Encoding: a data compression algorithm adapted to learn subword vocabularies.
Break it on purpose
If you type text with no known merges, the algorithm cannot combine characters into larger tokens.