The NLP Pipeline

Why Does This Exist?

Raw text resists direct math. "Tim Cook visited Berlin on Monday" is one string, yet a model needs to know where words start, which forms match, who did what, and which spans are names. Guessing from characters alone keeps every ambiguity alive at once.

The pipeline exists to remove ambiguity in order. Split first so later steps have units. Clean forms so "Visited" and "visited" match. Label grammar and names so the final model reads structure instead of soup. Modern large models fold many steps inside, but they still solve the same sub-problems, and debugging them still means checking each stage.

This page covers the classical order and what each stage adds. For the splitting step itself, see tokenization in NLP. For the grammar-labelling step, see part-of-speech tagging. For the name-span step, see named entity recognition. For who-did-what links between words, see dependency parsing.

The one idea you need first

A token is the unit later steps annotate, usually a word or subword. Without tokens there is nothing to tag, link or group. The full treatment is in tokenization in NLP.

Think of It Like This

How It Actually Works

Call the raw input $S$, a plain character string. Tokenization maps $S$ to $T = (t_1, \ldots, t_n)$, where each $t_i$ is one token. Normalization maps each $t_i$ to a standard form $u_i$: lowercasing folds case, lemmatization maps "visited" to "visit" by dictionary plus grammar, stemming chops endings fast but roughly, and stop-word removal drops glue words kept in a list. Tagging maps each kept token to a grammar label $p_i$ such as noun or verb. Parsing adds links $d_i$ such as subject-of between tokens. Entity labelling merges token runs into spans $(a, b, y)$ from position $a$ to $b$ with type $y$ such as person or place. The task model reads all of it and outputs $\hat{y}$, the prediction for the job at hand.

Order matters because each map needs the last one done: tags need tokens, parses need tags, spans need tokens plus grammar. Skipping or swapping stages feeds a step input it was never built to read.

From characters to tokens

Segmentation finds sentence ends without splitting "Dr." or "D.C.", then tokenization splits on rules or learned subwords. Normalization then shrinks variety: six raw pieces become five content pieces once the glue word leaves. Which cleaning to apply depends on the job, because the same fold that helps matching can erase a signal names need.

From tokens to structure

Tagging resolves "visited" as verb from context. Parsing links "Tim Cook" as the visitor and "Berlin" as the visited place. Entity labelling groups "Tim Cook" as one person span. Each layer adds one fact the final model would otherwise have to rediscover from characters.

Worked example

Take $S =$ "Tim Cook visited Berlin on Monday". Lowercased tokenization gives six pieces: ["tim", "cook", "visited", "berlin", "on", "monday"], so $n = 6$. Drop the stop word "on" and five content tokens remain: ["tim", "cook", "visited", "berlin", "monday"]. That is a $1 / 6 \approx 0.167$ cut before any model runs. Tagging then marks "visited" as verb, and span labelling merges positions 1 to 2 into ("Tim Cook", person). The same six-to-five drop is drawn in the middle of the diagram.

Code

This snippet repeats the worked token counts with only the standard library.

1import re2
3
4def tokens(text: str) -> list[str]:5    return re.findall(r"[A-Za-z]+", text.lower())6
7
8text = "Tim Cook visited Berlin on Monday"9t = tokens(text)10print(t, len(t))  # -> ['tim', 'cook', 'visited', 'berlin', 'on', 'monday'] 611kept = [w for w in t if w not in {"on"}]12print(kept, len(kept))  # -> ['tim', 'cook', 'visited', 'berlin', 'monday'] 5

Watch Out For

The Quick Version

• The pipeline maps raw string S to tokens T to cleaned forms to tags, links and spans, then to prediction.
• Each stage needs the last one done: tags need tokens, parses need tags, spans need both.
• The worked sentence gives six tokens and five content tokens after dropping "on", a one-sixth cut.
• Cleaning is per task: what helps matching can hurt name finding.
• Early errors stick downstream, so inspect stage outputs when the final score drops.

What to Read Next

• Language Detection
• Part-of-Speech Tagging
• Named Entity Recognition