Bag of Words
Each sentence becomes a tally of its words over one shared list, so sentences with similar tallies likely share a topic.
Why Does This Exist?
Models eat fixed-width numbers. Text arrives as variable-length strings, so you'll need a bridge that turns a sentence into a row of numbers before any classifier can touch it.
Counting is the oldest bridge and it still earns its keep. Fix one shared word list from your training data, after text preprocessing has lowercased and split the text, then tally each document against it. A 50,000-word list turns a 15-word note into a 50,000-wide row with only 15 nonzero entries. The bet is simple: topic shows in the tallies even after order is gone.
This page covers the counting step and how you compare two tallies with cosine similarity. TF-IDF is the direct sequel and weights each count by rarity so filler words stop dominating. Word2Vec and sentence embeddings learn meaning-aware vectors for the jobs counting can't do, and text classification is where you'll first put these tallies to work.
Think of It Like This
The pantry inventory
You run a shared kitchen and want to guess what each cook made without watching them. So you fix one shelf list: flour, sugar, salt, chili, fish sauce. After each session you count the jars back onto the shelf and write down the tally. One cook reads 3, 0, 1, 0, 2. Another reads 3, 0, 1, 2, 0. You can taste the overlap before anyone describes a dish.
The list is the vocabulary, each tally is the count vector, and the shelf order never mattered. You shuffled the jars and kept the counts.
Where it stops: a tally can't tell fish sauce added early from fish sauce added late, and it can't tell "no chili" from "extra chili". When sequence or negation decides the outcome, the inventory guesses wrong and you need a method that reads order.
How It Actually Works
Fix a vocabulary , meaning one ordered list of every distinct token in your training text. Each document becomes a count vector , meaning a row of numbers with one entry per vocabulary word, where entry holds how many times word appears in . The width is , the size of the list, and it never changes. That fixed width is the whole trick: variable-length sentences in, same-width tallies out.
Where does the shared list come from?
Collect your training corpus, run it through text preprocessing, gather every distinct token, and freeze that order. Our two-sentence corpus gives as [cats, chase, dogs, mice], so and every vector from here on is four wide. A word that never appeared in training has no column, so at test time it's dropped silently. That dropped word is called out of vocabulary, and a large gap between training words and live words is the first sign your list is stale.
How do sentences become tallies?
Start from zeros and bump one column per token. Sentence A, "cats chase mice", touches cats once, chase once, dogs never, mice once:
Sentence B, "dogs chase cats", touches cats once, chase once, dogs once, mice never:
A repeat would bump past 1, so "cats chase cats" would read . Order never enters the procedure, which is why "mice chase cats" tallies exactly like A. In production the same idea runs at scale: a 50,000-word vocabulary turns a 15-word note into a row with 49,985 zeros, a sparse vector you store in a compressed format instead of a dense array.
Worked example: one cosine from two tallies
Cosine similarity asks how parallel two tallies point, on a scale where means identical direction and means no overlap. Write for the dot product (multiply matching entries, then add) and for the length (square root of the sum of squares):
The dot product counts the shared weight:
Each length covers three ones:
So the product of the lengths is and:
The diagram draws these same numbers: two sentences, two four-wide tallies, one 0.67. Two shared words out of three pushes the pair well above zero without reaching 1.0, which is reserved for identical tallies. Cosine is the right ruler here because it divides out length: a long review and a short one on the same topic point the same way even when their raw counts differ.
Code
The point is the check: build the vocabulary, read off the two tallies, and confirm the hand computation of 0.67.
import numpy as npfrom sklearn.feature_extraction.text import CountVectorizer
corpus: list[str] = ["cats chase mice", "dogs chase cats"]vectorizer: CountVectorizer = CountVectorizer()counts: np.ndarray = vectorizer.fit_transform(corpus).toarray()
print(vectorizer.get_feature_names_out()) # -> ['cats' 'chase' 'dogs' 'mice']print(counts[0]) # -> [1 1 0 1]print(counts[1]) # -> [1 1 1 0]
dot: float = float(counts[0] @ counts[1])product: float = float(np.linalg.norm(counts[0]) * np.linalg.norm(counts[1]))print(f"{dot:.0f} {product:.2f} {dot / product:.2f}") # -> 2 3.00 0.67The printout matches the hand computation above: vocabulary [cats, chase, dogs, mice], tallies and , dot product 2, length product 3.00, cosine 0.67.
Watch Out For
Twins with opposite meanings
Bag of words gives "cats chase mice" and "mice chase cats" the identical tally . Negation breaks the same way: "good" and "not good" share their "good" count.
Symptom: a classifier that aces topic sorting yet fails every order-sensitive check, like sentiment flips or subject swaps. Fix: add bigrams so "not good" gets its own column for a quick patch, then move to a sequence-aware model such as Word2Vec context windows or sentence embeddings when order decides the label.
Filler words setting the score
Raw counts let frequent glue words vote the loudest. Two unrelated reviews can score 0.9 cosine on shared "the", "and", "is" while their content words barely register.
Symptom: everything looks similar to everything, and removing ten stopwords moves your accuracy more than tuning the model. Fix: cut stopwords in text preprocessing and switch the weighting to TF-IDF, which discounts words that appear in nearly every document.
The Quick Version
- Bag of words turns each document into a fixed-width tally over one shared vocabulary, dropping order entirely.
- Our pair "cats chase mice" and "dogs chase cats" becomes and over the four-word list [cats, chase, dogs, mice].
- Cosine similarity of those tallies is 2 divided by 3, about 0.67, with 1.0 reserved for identical tallies.
- Real tallies are sparse: a 15-word note against a 50,000-word vocabulary is almost entirely zeros.
- Words outside the training vocabulary are dropped, and swapped word order is invisible, so order-sensitive jobs need richer representations.