Text Normalization

Why Does This Exist?

Humans read Run, RUN, running and ran as one idea without thinking. A model with a fixed vocabulary does not. Each surface form gets its own row, its own vector, and its own training bill. The signal splits four ways, rare forms barely get trained, and the vocabulary balloons with entries that teach the same thing.

Normalization exists to collapse those surface differences into one canonical form before tokenization and learning. Four variants become one lemma, three duplicate rows disappear, and every example trains the same entry. It trades a little upfront rule writing for a smaller table and a stronger signal per entry.

This page covers the cleaning pass only. The cut into pieces lives in tokenization, the full order of steps in the NLP pipeline, and the grammar step that needs clean tokens in part of speech tagging.

The one idea you need first

A token is the unit the model counts, often a word or subword with its own ID. If Run and run map to two IDs, the model learns them separately. Normalization decides which differences to erase so one meaning maps to one ID.

Think of It Like This

How It Actually Works

Normalization is a short pipeline of fixed rules, applied in order. Each rule erases one kind of variation. Use only the rules your task can afford.

Lowercase and fold

Lowercasing maps RUN to run and The to the. Case folding goes further for special letters, such as mapping the German sharp s to ss so it matches the expected spelling. One line, huge dedup. Skip it when case carries meaning, such as names in entity tagging.

Expand and strip

Contractions pack two words into one. A small dictionary maps short forms back to two words, turning do not written as one token into do and not, so the negation stands alone where the model can weigh it. Then strip what adds no signal for your task: commas that split dog from dog, links, tags and extra spaces. Keep what your task reads. Code keeps punctuation, prose often does not.

Reduce to base form

This is the morphological core. Stemming chops endings with blind rules, so running becomes run but ponies becomes poni. Fast, crude, no dictionary. Lemmatization looks the word up and returns its dictionary form, so better becomes good and ran becomes run. It needs the grammar tag to choose well, since saw can come from see or name a tool.

Worked collapse

Take four surface forms: Run, RUN, running, ran.

Lowercase: Run becomes run, RUN becomes run, running stays running, ran stays ran. Distinct forms drop from 4 to 3.

Lemmatize as verbs: run stays run, running becomes run, ran becomes run. Distinct forms drop from 3 to 1.

Four rows become one lemma run. Three duplicate entries disappear, and every later example of any variant trains the same row. Those are the numbers in the diagram above.

The trade is plain. Aggressive cleaning buys a small table and dense counts, and it sells off distinctions your later step may need. Modern fluent models clean less and keep stop words, because word order and small words carry the structure they generate from.

Code

The collapse from the worked example, with plain rules so the 4 to 3 to 1 counts can be checked.

1import re2
3
4def normalize(tokens: list[str]) -> list[str]:5    lowered: list[str] = [t.lower() for t in tokens]6    lemma_map: dict[str, str] = {"running": "run", "ran": "run"}7    return [lemma_map.get(t, t) for t in lowered]8
9
10raw: list[str] = ["Run", "RUN", "running", "ran"]11print(raw)  # -> ['Run', 'RUN', 'running', 'ran']12print(len(set(raw)))  # -> 413mid: list[str] = [t.lower() for t in raw]14print(mid)  # -> ['run', 'run', 'running', 'ran']15print(len(set(mid)))  # -> 316done: list[str] = normalize(raw)17print(done)  # -> ['run', 'run', 'run', 'run']18print(len(set(done)))  # -> 1

Four distinct forms, then three after lowercasing, then one after mapping running and ran to run.

Watch Out For

The Quick Version

• Text normalization maps surface variants to one canonical form before tokenization.
• Lowercasing and folding erase case differences in one pass.
• Expanding contractions frees negations, and stripping removes marks your task does not read.
• Stemming chops blindly while lemmatization looks words up, so ran becomes run only in the second.
• Worked case: Run, RUN, running and ran collapse from 4 forms to 3 to 1 lemma.
• Clean for the task at hand, since each rule sells off a distinction some later step may need.

What to Read Next

• Spell Correction
• Tokenization in NLP
• Part-of-Speech Tagging