Skip to content
AI360XpertBeta

Text Preprocessing

Models count numbers, not words. Preprocessing turns a messy sentence into clean base-form tokens, so Cat, cat and cats all count as one word.

One raw sentence becomes three clean tokens after lowercasing and splitting and dropping filler words
One raw sentence becomes three clean tokens after lowercasing and splitting and dropping filler words

Why Does This Exist?

A classifier eats fixed-width rows of numbers. Text arrives as variable-length strings, so something has to bridge that gap before any model can touch a sentence. Feed the raw string straight in and the pipeline crashes on the first punctuated word.

Raw text is also noisy in a way that silently wrecks counting. Capital letters split one word into two entries, punctuation glues itself to words, and filler words like the and are drown out the content. Skip this step and Cat, cat and cats become three strangers in your vocabulary, the word list balloons, and rare telling words get buried under grammatical glue.

This page covers the cleaning and splitting step only. Turning those clean tokens into numbers belongs to bag of words, and weighting them by rarity belongs to TF-IDF. Both assume the text arriving at their door is already clean.

Think of It Like This

Sorting a jar of mixed foreign coins

Tip a jar of mixed coins on a table and you do four things before counting anything. You rinse off the grime, you flip every coin face up so the same coin looks the same twice, you sort them into trays by denomination, and you toss the washers and tokens that are not currency at all. Only then do you count.

Preprocessing is that table. Rinsing is stripping punctuation, flipping face up is lowercasing, sorting into trays is splitting into tokens, and tossing washers is dropping filler words. Grouping cats with cat is deciding two worn coins share one slot.

Where it stops: a coin keeps its value however it lands, but a word can lose meaning when you normalize it. Apple the company and apple the fruit are different things, and saw as a noun is not saw as a verb. Every normalization step trades a smaller vocabulary for a little lost meaning, so you check that trade per task.

How It Actually Works

Classical preprocessing is a short pipeline, and order matters because each step assumes the last one ran.

Lowercase and strip

First the surface cleanup. Lowercasing merges Cat and cat into one entry, and stripping punctuation detaches quickly! from quickly. This is also where you remove the artifacts of your data source: markup, links, extra whitespace. The sentence The cats are running quickly! becomes the cats are running quickly.

Split into tokens

Tokenization cuts the string into pieces. The classical cut is whitespace, so the cleaned sentence becomes five tokens: [the, cats, are, running, quickly]. Five strings, five future vocabulary entries.

Drop the filler, keep the base form

Filler words carry grammar, not topic, so a small stopword list removes the and are, leaving [cats, running, quickly]. Then lemmatization maps each survivor to its dictionary base: cats to cat, running to run, quickly stays quickly. Final output: [cat, run, quickly].

Worked example

Count the shrinkage on that one sentence. Raw distinct words: 5 (The, cats, are, running, quickly). After lowercasing and splitting: 5 tokens. After dropping 2 filler words: 3 tokens. After lemmatization: [cat, run, quickly], still 3 tokens but now cats and any future cat merge into one column instead of two. On a whole corpus that merge repeats thousands of times, which is where the vocabulary savings actually come from.

Code

The point is the check: run the pipeline on one sentence and confirm the exact list that comes out, so a silent change in any step shows up here first.

import string

def preprocess(text: str) -> list[str]:    lowered = text.lower()  # Cat and cat become one entry    cleaned = lowered.translate(str.maketrans("", "", string.punctuation))    tokens = cleaned.split()  # whitespace cut into pieces    kept = [t for t in tokens if t not in {"the", "are"}]  # drop filler    lemma = {"cats": "cat", "running": "run"}  # tiny stand-in dictionary    return [lemma.get(t, t) for t in kept]

print(preprocess("The cats are running quickly!"))# -> ['cat', 'run', 'quickly']

Three tokens out, matching the hand-traced example above. A production setup swaps the toy dictionary for a real lemmatizer, but the shape of the pipeline stays the same.

Watch Out For

Lowercasing text that feeds a subword model

Symptom: you clean text for a transformer the classical way, and accuracy on names and entities drops. BERT-style models split words into pieces and read case and punctuation as signal, so Apple and apple genuinely differ to them and stopwords carry the grammar they need.

Fix: match the cleaning to the consumer. Classical counters get the full pipeline; subword models get raw text with only the cleaning their tokenizer docs ask for. When in doubt, feed the model less cleaning, not more.

Deleting a word that carries the label

Symptom: sentiment accuracy collapses after you add stopword removal, and inspection shows not good became good. Negations, question words and domain terms often sit on the default stopword list while carrying the exact signal your task needs.

Fix: never trust the default list blindly. Keep negations like not and no for sentiment work, review the list against your most predictive unigrams, and rerun the check in Code above after every list change. Inspection of a few flipped examples usually names the culprit word directly.

The Quick Version

  • Models need numbers, so text must be cleaned and split into tokens before counting starts.
  • Lowercasing and punctuation stripping merge surface variants into single entries.
  • Tokenization cuts the string into pieces, classically one per whitespace-separated word.
  • Stopword removal drops grammatical filler, and lemmatization maps variants like cats to one base form.
  • The full classical cleaning suits counters, but subword models want raw text with its case and grammar intact.