TF-IDF
Common words shout the loudest in raw counts. TF-IDF turns down words that appear everywhere and turns up words that are frequent here but rare elsewhere.
Why Does This Exist?
Counting words has a loud flaw. In an article about quantum physics the most frequent words are not quantum or entanglement but the, is and of. Rank documents by raw counts and a query for the physics returns whatever says the the most, burying the actual topic.
TF-IDF fixes the ranking with one idea: a word matters when it is frequent in this document and rare across all documents. Local frequency says the word describes this page; global rarity says the word distinguishes this page from the rest. Multiply the two and filler cancels itself while telling words rise.
This page covers the weighting step that sits on top of counting. The counting itself lives in bag of words, and the dense vectors that understand synonyms live in Word2Vec.
Think of It Like This
The toolbox test
Guess a worker's trade from their tools. A pen tells you nothing: doctors, lawyers and teachers all use one daily, so it is frequent everywhere and discriminates nowhere. A stethoscope tells you everything: used a few times a day, but almost never outside medicine, so it is locally frequent and globally rare.
TF-IDF scores tools exactly that way. The pen gets a high local count times a near-zero rarity, landing near zero. The stethoscope gets a modest count times a high rarity, landing on top.
Where it stops: a rare tool can still be irrelevant. A plumber's torch in a dentist's office is rare everywhere yet says nothing about dentistry. Rarity alone is not relevance, which is why the local-frequency half of the product has an equal vote.
How It Actually Works
Two counts multiply into one score. Call the total document count and the number of documents holding the word .
Term frequency
How much of this document is the word:
is a share, so long and short documents compare fairly.
Inverse document frequency
How rare the word is everywhere else:
The log keeps a word that appears in 1 of 1,000 documents from outscoring everything else a thousandfold. When a word sits in every document, equals 1 and the log equals 0, so the final score is 0 no matter the local count.
Worked example
Four short documents: cat sat on mat, dog sat on log, cat chased dog, and quantum cat sat quantum theory rocks. Here .
Take quantum in document 4. It appears 2 of 6 words, so which is about 0.333. It appears in 1 of 4 documents, so which is about 1.386. The product is which is about 0.462.
Take sat in document 1. It appears 1 of 4 words, so . It appears in 3 of 4 documents, so which is about 0.288. The product is which is about 0.072.
Same local story, one appearance each, yet quantum outscores sat more than sixfold because sat is shared across the collection and quantum belongs to one page. That gap is the whole point of the weighting.
Code
The point is the reproduction: plain counting plus one log call lands on the hand-computed pair.
import math
def tfidf(corpus: list[str], doc_id: int, word: str) -> float: toks = [d.split() for d in corpus] tf = toks[doc_id].count(word) / len(toks[doc_id]) df = sum(1 for t in toks if word in t) return tf * math.log(len(corpus) / df)
corpus = [ "cat sat on mat", "dog sat on log", "cat chased dog", "quantum cat sat quantum theory rocks",]
print(f"{tfidf(corpus, 3, 'quantum'):.3f}")# -> 0.462print(f"{tfidf(corpus, 0, 'sat'):.3f}")# -> 0.072The rare word wins by the same margin as the worked example, which is how you know both are right.
Watch Out For
Reading not good as good
Symptom: a review classifier trained on TF-IDF vectors rates not good as praise. The vector counts good once and drops the order, so negation, intensifiers and sarcasm all vanish before the model sees them.
Fix: add word pairs alongside single words so not good becomes its own feature, or move to vectors that read order. Counting tricks cannot recover sequence once it is gone.
Treating car and automobile as strangers
Symptom: two documents about the same vehicle score near-zero similarity because one says car and the other says automobile. TF-IDF gives every distinct spelling its own axis, so synonyms are as distant as unrelated words.
Fix: use this weighting for keyword search and first baselines, then switch to Word2Vec when meaning has to survive paraphrase. Statistics over spellings can never learn that two spellings share one thing.
The Quick Version
- Raw counts let filler words dominate, so search and classifiers drown in the and of.
- Term frequency measures a word's share inside one document, keeping lengths comparable.
- Inverse document frequency takes the log of total documents over documents holding the word, so ubiquitous words score zero.
- The product rewards words that are frequent here and rare elsewhere, as 0.462 against 0.072 shows.
- Order blindness and synonym splitting stay broken, and dense vectors are the fix for both.