Skip to content
AI360XpertBeta
Core ML

Visual explainer

Tokenization

Trace a counted BPE corpus from characters to reusable pieces, IDs and decoding, including vocabulary costs and lossy normalization.

Tokenization turns text into pieces and vocabulary IDs.
Tokenization turns text into pieces and vocabulary IDs.

A tokenizer divides text into reusable pieces and assigns each piece an integer ID. The model uses those IDs to select numerical representations. A token can be part of a word, a whole word, punctuation or a byte sequence, depending on the tokenizer.

Our synthetic tokenizer encodes abac as aba | c, with IDs [4, 2]. These are invented teaching strings and IDs, not a claim about a production model. The next scenes show exactly how this vocabulary is learned.

Look Up the Representation

Token IDs select embedding rows rather than encode magnitude.
Token IDs select embedding rows rather than encode magnitude.

The complete toy vocabulary is a:0, b:1, c:2, ab:3, aba:4. An ID is an address, not a semantic measurement: ID 4 isn't twice the meaning of ID 2. The selected row illustrates a separate embedding table: row 4 returns (1, 2), while row 2 returns (−1, 0). These vectors are synthetic too.

The tokenizer supplies discrete IDs; the embedding layer supplies floating-point vectors. See word embeddings for the representation step.

Count Adjacent Pairs

The most frequent adjacent pair wins this BPE training round.
The most frequent adjacent pair wins this BPE training round.

Byte Pair Encoding (BPE) learns a sequence of merges. Our corpus contains ab three times, aba twice and ac once. Start with individual characters. The pair (a,b) occurs 3 + 2 = 5 times, (b,a) occurs twice, and (a,c) once. Equal-scale bars show these weighted counts. The winner is (a,b).

This follows the pair-counting mechanism in Sennrich et al., section 3.2. Our simplified variant treats each string as a separate unit, blocks cross-unit merges and omits the paper's end-of-word symbol. Nothing in the example merges across corpus entries.

Remove One Boundary

Merging a with b removes a boundary in each matching pair.
Merging a with b removes a boundary in each matching pair.

In the pictured aba occurrence, a | b | a becomes ab | a. The unchanged last a makes the removed boundary visible. Apply that merge throughout the corpus: ab×3, ab | a×2, a | c×1. The token total falls from 3×2 + 2×3 + 1×2 = 14 to 3×1 + 2×2 + 1×2 = 9.

Recount the remaining adjacent pairs. (ab,a) has count 2 and (a,c) count 1, so the second merge creates aba. The total becomes 3×1 + 2×1 + 1×2 = 7 tokens. Subword units follow corpus statistics; they needn't be meaningful prefixes or grammatical roots.

Count Attention Positions

Fewer tokens reduce the number of dense attention positions.
Fewer tokens reduce the number of dense attention positions.

For held-out abac, four character tokens become two subword tokens, aba | c. A full, unmasked attention grid has one position for every ordered token pair: 4² = 16 versus 2² = 4. Cell size is identical in both grids.

That is a 75% reduction in these grid positions, not a measured 75% speedup. Causal masking permits 10 versus 3 positions in the lower triangle including its diagonal; kernels may still compute or store different amounts. Projections, feed-forward layers, decoding and hardware also affect total cost. Shorter sequences can fit more text within a fixed token budget, with gains depending on the text and vocabulary.

Pay for the Vocabulary

Extra vocabulary rows trade table storage for shorter sequences.
Extra vocabulary rows trade table storage for shorter sequences.

The two learned pieces add two rows: vocabulary size rises from 3 to 5. With an embedding width of two, the table grows from 3×2 = 6 to 5×2 = 10 scalar parameters. The mint outline identifies the new ab and aba rows. Alongside that storage increase, this example's sequence drops from four tokens to two.

Output projections may also grow with vocabulary size; weight tying affects how many distinct parameters are stored. A larger vocabulary doesn't guarantee whole words, better language coverage or a fixed compression factor. Evaluate on the intended corpus. The toy size excludes special tokens, whitespace and byte fallback.

Decode the Pieces

Decoding concatenates this toy vocabulary's pieces.
Decoding concatenates this toy vocabulary's pieces.

For this single-unit, unchanged-text example, ID 4 retrieves aba, ID 2 retrieves c, and concatenation recovers abac. Different ID sequences can decode to the same string: [0,1,0,2] also spells abac. Reversible piece lookup doesn't make the entire encoding pipeline bijective.

Token boundaries aren't character boundaries. A model can learn spelling information from training; subword IDs don't make character counting impossible. Nor does adding hyphens guarantee single-character tokens or correct arithmetic. Inspect the actual encoding when character structure matters.

Apply Frozen Merge Rules

Inference applies learned merges without recounting the prompt.
Inference applies learned merges without recounting the prompt.

The unseen string abac starts as a | b | a | c. Apply the learned first rule to obtain ab | a | c, then the second to obtain aba | c. The tokenizer doesn't relearn frequencies from this prompt or create a new abac vocabulary entry. The vocabulary remains five pieces.

As a fresh check, ac remains [0,2]: neither learned merge applies. New characters outside this toy alphabet are unsupported; practical coverage depends on the chosen base alphabet, unknown-token policy or byte fallback.

Respect Pretoken Boundaries

A pretoken boundary blocks a merge across separate units.
A pretoken boundary blocks a merge across separate units.

Now consider ab a, containing a space. If pretokenization makes ab and a separate units, the learned ab+a rule cannot cross the dashed boundary. The stopped bridge pictures that blocked merge. This is a boundary illustration, not a complete encoding of the space with our five-entry vocabulary.

Real pipelines specify whitespace handling and any boundary markers. Some preserve spaces as bytes or piece prefixes; others normalize or discard them. Use the tokenizer matched to the trained model. The Hugging Face pipeline documentation distinguishes normalization, pretokenization, token modeling and postprocessing.

Where It Breaks

Lowercasing can make different original strings share one encoding.
Lowercasing can make different original strings share one encoding.

Add an optional lowercase normalizer before our BPE rules. Both ABAC and abac become abac, then [4,2]. Decoding returns lowercase text and cannot determine the original case from these IDs alone. The failure occurs before piece lookup: normalization discarded a distinction.

Use a pipeline that preserves the distinctions your task needs, and keep original text or offsets when necessary. Lossless tokenizers exist; exact reconstruction is a property to verify, not a promise made by all tokenization schemes.

The Quick Version

  • Tokenizers produce pieces and IDs; embeddings provide numerical representations.
  • BPE learns frequent adjacent-pair merges from a training corpus.
  • Inference reuses learned merges and vocabulary.
  • Shorter sequences reduce attention positions, with task-dependent runtime gains.
  • Extra vocabulary rows cost storage.
  • Pretoken boundaries constrain which merges can apply.
  • Normalization can destroy information before token lookup.