The NLP Pipeline

Why Does This Exist?

Raw human language is noisy, ambiguous, and inconsistent. A sentence like "the bank can guarantee deposits" means different things depending on context. The NLP pipeline converts raw text into representations where a model can do useful work, by progressively reducing ambiguity at each step.

The classical pipeline emerged in the 1990s, when chaining a POS tagger before a parser before an information extractor was the only way to make progress. Each step narrowed ambiguity enough for the next step to operate. Modern LLMs do many of these steps implicitly, but understanding the pipeline tells you exactly what problem each component solves.

Think of It Like This

How It Actually Works

Stage 1: Sentence segmentation

Before processing text, you need to know where sentences end. Splitting on periods fails for "Dr. Smith visited Washington D.C." A sentence segmenter uses rules or a trained model to distinguish sentence-ending periods from abbreviation periods.

Stage 2: Tokenization

A token is the smallest unit the pipeline processes — usually a word or subword. Splitting on whitespace fails for "New York", "don't", and "U.S.A.". A tokenizer applies language rules or learned subword algorithms (like BPE) to produce a clean token sequence.

Stage 3: Normalization

Normalization reduces token variety:

• Lowercasing: "Apple" → "apple"
• Lemmatization: "running" → "run" (dictionary lookup + grammar)
• Stemming: "running" → "runn" (suffix stripping — fast but crude)
• Stop-word removal: drop tokens like "the", "is", "at"

Which normalizations to apply depends on the downstream task.

Stage 4: Part-of-speech tagging

Each token gets a grammatical label: noun, verb, adjective, etc. Context resolves ambiguity: "Time flies like an arrow" vs "Fruit flies like a banana" — "flies" is a verb in the first and a noun in the second.

Stage 5: Parsing

Parsing determines sentence structure. Constituency parsing groups tokens into nested phrases (NP, VP). Dependency parsing labels relationships between token pairs (subject-of, object-of). Downstream relation extraction relies on parse structure to know what modifies what.

Stage 6: Named entity recognition

NER labels spans of tokens as real-world entities: people, organizations, locations, dates. "Barack Obama visited Berlin" becomes [PER Barack Obama] visited [LOC Berlin].

Stage 7: Task-specific model

With text structured into sentences, tokens, POS tags, parse trees, and entities, the final step applies the task model: a sentiment classifier, a relation extractor, a machine translation encoder.

Code

1import spacy2
3nlp = spacy.load("en_core_web_sm")4
5text = "Apple's CEO Tim Cook visited Berlin on Monday."6doc = nlp(text)7
8print("Tokens and POS:")9for token in doc:10    print(f"  {token.text:<15} {token.pos_:<8} {token.dep_}")11
12print("\nNamed entities:")13for ent in doc.ents:14    print(f"  {ent.text:<20} {ent.label_}")

Output:

1Tokens and POS:2  Apple          PROPN    nsubj3  's             PART     case4  CEO            NOUN     appos5  Tim            PROPN    nmod6  Cook           PROPN    nmod7  visited        VERB     ROOT8  Berlin         PROPN    dobj9  on             ADP      prep10  Monday         PROPN    pobj11  .              PUNCT    punct12
13Named entities:14  Apple                ORG15  Tim Cook             PERSON16  Berlin               GPE17  Monday               DATE

Watch Out For

The Quick Version

• The NLP pipeline converts raw text into progressively structured representations: sentences → tokens → POS tags → parse trees → entities → task predictions.
• Each step reduces ambiguity enough for the next step to operate reliably.
• Modern LLMs perform many steps implicitly, but the pipeline framing tells you what each component is responsible for.
• Apply normalizations relative to your downstream task — not all normalizations help all tasks.