Data Concepts

Text Data Preparation

Classical text cleaning and a pretrained tokeniser want opposite things. Strip the casing, punctuation and stopwords for one, and you damage the other.

One messy string runs through Unicode normalisation, noise removal and a subword tokeniser, with lowercasing and stopword removal drawn off to one side because they belong to the bag-of-words route and nowhere else

Why Does This Exist?

A pipeline you inherited does four things to every document: lowercase it, strip the punctuation, drop the stopwords, stem what's left. Then it hands the result to a pretrained transformer.

All four make that model worse.

None of them are wrong in themselves. In a bag of words the vocabulary is the column list, so Dog, dog and dogs in three columns splits the evidence three ways. Fold them into one and each column gets more.

A subword tokeniser inverts the deal. It was pretrained on raw text with casing, punctuation and stopwords left in, and its merge rules are a record of what that text looked like. Flatten your input and you hand it strings the pretraining distribution never contained.

So the rule is blunt. Clean hard for a bag of words. For a pretrained model, clean almost nothing beyond genuine noise.

Think of It Like This

Two people reading the same letter

A complaint arrives in the post, and two people have to do something with it.

The first runs a sorting desk. She never reads a letter, she tallies it: how many mention "refund", how many mention "delivery". Retype that letter in flat lowercase with the punctuation gone and the filler words dropped, and her afternoon gets shorter.

The second knows this customer by name. He reads the whole thing, and the parts you deleted are the parts he uses: the capitals on a product name, the question mark that turns a remark into a demand, the word "never" sitting in the second paragraph.

Hand him the retyped version and he'll still write back. He'll just answer the wrong letter, confidently.

How It Actually Works

Order matters as much as the steps.

Encoding and Unicode, before anything else

Decode the bytes as UTF-8 and check you got language back. Read UTF-8 as Latin-1 and é arrives as Ã©, now a token in your vocabulary.

Then normalise the form. One visible character can be two byte sequences: é is one code point (U+00E9) or two (e plus a combining acute). Same glyph, both valid, and == says they differ, so deduplication misses the pair and a lookup misses the word. Pick NFC, apply it once at the boundary, put it in your data contract.

Then the characters you can't see. A zero-width joiner (U+200D) splits a word in two, a non-breaking space (U+00A0) looks like a space and isn't one, so split(" ") keeps it glued to the next word, and four horizontal strokes all get called a dash. One appearance, four hashes.

Then real noise, then language

Strip what isn't language: HTML tags, boilerplate headers and footers, navigation chrome, the signature block repeated across 400,000 emails. Boilerplate is the expensive one. If a third of every scraped page is the same nav block, that's a third of what the model reads.

Identify the language next, before anything language-specific runs. A corpus that's 8% Portuguese poisons a monolingual English model quietly, and it breaks stemming, which assumes English suffixes.

The steps that belong to a bag of words only

Casing, punctuation, stopwords, stemming or lemmatisation. All four shrink a vocabulary, which is the point when your columns are the vocabulary.

Stemming chops suffixes by rule, crudely by design: the Porter stemmer maps university and universe to the same univers. Lemmatisation returns dictionary forms, so better becomes good, but it needs the part of speech and runs slower.

Tokenisation, where the real decisions are

Word-level tokenisation splits on whitespace and punctuation, so the vocabulary grows with the corpus, never covers it, and every unseen word collapses to one unknown symbol.

Character-level has no unknowns and a vocabulary of a few hundred, but sequences get five or six times longer.

Subword sits between them and won. BPE, WordPiece and SentencePiece learn frequent pieces from a corpus, then decompose anything unseen into pieces they know: tokenising becomes token plus ising. That's the whole answer to out-of-vocabulary text.

And token count is not word count. Merges learned on mostly-English prose mean an unfamiliar script pays more per character, so a Hindi or Thai paragraph can cost double its English translation, and JSON burns tokens on braces.

Truncation, and why chunking usually wins

Cut a document at the context length and you keep the beginning. A support ticket ends "so I'm cancelling", which is the label, and truncation just deleted it. Chunk with a small overlap and pool the predictions instead.

Worked example

Take " <p>Café RESERVATIONS are NOT open yet!!</p> ", with Café arriving in the two-code-point form.

Normalise and strip the noise and you get Café RESERVATIONS are NOT open yet!!: 37 code points in one Unicode form, 36 in the other, equality false, six whitespace words.

Classical route, three tokens left: café, reservations, open. The default English stopword list in scikit-learn holds not, no and never, so a string that said reservations are not open now says they're open. The sentiment flipped and nothing reported it.

Pretrained route, twelve pieces under one plausible segmentation: Caf, é, RES, ERV, ATIONS, are, N, OT, open, yet, !, !. Twelve tokens from six words, the all-caps ones costing most.

Show Me the Code

1import unicodedata2
3raw: str = "Caf\u0065\u0301 RESERVATIONS are NOT open yet!!"  # e plus a combining acute4stops: frozenset[str] = frozenset({"are", "not", "yet"})      # all three are on the default list5
6
7def classical(text: str) -> list[str]:8    """Lowercase, drop punctuation, drop stopwords: the bag-of-words route."""9    words = "".join(c if c.isalnum() or c == " " else " " for c in text).split()10    return [w for w in (w.lower() for w in words) if w not in stops]11
12
13nfc: str = unicodedata.normalize("NFC", raw)14print(len(raw), len(nfc))                     # -> 37 3615print(raw == nfc)                             # -> False16print(classical(nfc))                         # -> ['café', 'reservations', 'open']17print(len(nfc.split()), len(classical(nfc)))  # -> 6 3

Two lines carry the page. The first says one visible string has two lengths. The third says your cleaning deleted the negation.

Watch Out For

Lowercasing and dropping stopwords in front of a pretrained tokeniser

Muscle memory from every text tutorial written before 2018, and it costs accuracy on the task you were trying to improve.

Stopwords first, because that damage is measurable. The default English list in scikit-learn holds not, no and never. Strip it and "the fix never worked" becomes "fix worked", while the label stays put.

Casing carries entity information: Apple and apple, US and us, IT and it. And the merges were fitted on text with all of this present, so a flattened string segments into more pieces and stranger ones.

Keep the two routes apart in code: one function for the bag-of-words path, one for the pretrained path, nothing shared but normalisation and noise removal.

Preprocessing that differs between training and serving

The transform you applied at training has to be the transform you apply at serving, character for character. Text gives you a dozen ways to break that.

An extra .strip() in the serving path. NFC at training and whatever the request body held at serving. A tokeniser library up a minor version with its normalisation changed. Each shifts the token sequence for the same sentence, and none raises anything.

So the model that evaluated fine now runs a point or two worse, with nothing odd in the logs. Nobody investigates, because nothing broke.

Ship the preprocessing as one function versioned with the model artefact, pin the tokeniser exactly, and log token counts on both sides. Online/offline skew is the general shape of this.

The Quick Version

Two routes, opposite instincts: a bag of words wants a shrunken vocabulary, a pretrained tokeniser wants raw text.
Unicode normalisation comes first. One visible character, two byte sequences, and equality says they differ.
Invisible characters do real damage: zero-width joiners, non-breaking spaces, four dashes that hash four ways.
Strip genuine noise, count boilerplate as noise, then identify the language before any language-specific step.
Casing, punctuation, stopwords and stemming belong to the classical route only. Stemming maps university and universe together.
Subword tokenisation ends the out-of-vocabulary problem, and token count is not word count: unfamiliar scripts and code pay more per character.
Truncation keeps the beginning when the label is usually at the end, so chunk with overlap.
not, no and never are stopwords by default, which has quietly ruined a lot of sentiment models.