Skip to content
AI360Xpert
Core ML

TF-IDF

A statistical measure that evaluates how relevant a word is to a document in a collection, rewarding frequent words but penalizing words that appear everywhere.

TF-IDF multiplies a word's local frequency by its global rarity, dampening the impact of common words like 'the' while highlighting unique keywords.
TF-IDF multiplies a word's local frequency by its global rarity, dampening the impact of common words like 'the' while highlighting unique keywords.

Why Does This Exist?

Bag of Words has a fatal flaw: it assumes that the most frequent word in a document is the most important word.

If you count the words in an article about quantum physics, the most frequent words will not be "quantum" or "entanglement". They will be "the", "is", and "of". If a search engine relies purely on word counts, a query for "the physics" will return documents that just happen to say "the" a lot, drowning out the actual topic.

TF-IDF (Term Frequency - Inverse Document Frequency) solves this by scaling the frequency of a word by how rare it is across the entire dataset. It mathematically boosts words that are unique to a specific document and suppresses words that are everywhere.

Think of It Like This

Think of It Like This

Imagine you are trying to guess the profession of someone based on the tools they use.

If they use a pen, it doesn't help you much. A doctor uses a pen, a lawyer uses a pen, a teacher uses a pen. The pen is very frequent, but it provides zero distinguishing information.

But if they use a stethoscope, you instantly know they are a doctor. A stethoscope might only be used a few times a day, but it is extremely rare across all other professions.

TF-IDF is the mathematical version of this logic. It scores "stethoscope" highly because it is unique to the doctor's document, and scores "pen" near zero because every document has one.

How It Actually Works

TF-IDF is the product of two separate metrics.

1. Term Frequency (TF)

How often does the word appear in this specific document? The simplest formula is just the raw count of the word divided by the total number of words in the document.

TF(w,d)=count of word w in document dtotal words in document d\text{TF}(w, d) = \frac{\text{count of word } w \text{ in document } d}{\text{total words in document } d}

If "quantum" appears 5 times in a 100-word document, its TF is 0.05.

2. Inverse Document Frequency (IDF)

How rare is the word across all documents? We calculate this by taking the logarithm of the total number of documents divided by the number of documents containing the word.

IDF(w)=log⁡(Nnumber of documents containing w)\text{IDF}(w) = \log\left(\frac{N}{\text{number of documents containing } w}\right)

  • If the word "the" appears in all 1,000 documents out of 1,000: log⁡(1000/1000)=log⁡(1)=0\log(1000/1000) = \log(1) = 0. The IDF is zero.
  • If the word "quantum" appears in only 10 documents out of 1,000: log⁡(1000/10)=log⁡(100)≈4.6\log(1000/10) = \log(100) \approx 4.6. The IDF is high.

The Final Score

We multiply them together: TF × IDF.

  • For "the": TF * 0 = 0. The word is completely neutralized, no matter how many times it appeared in the document.
  • For "quantum": 0.05 * 4.6 = 0.23. The word gets a high score because it is locally frequent but globally rare.

Show Me the Code

In Python, scikit-learn provides TfidfVectorizer, which does the counting and the math in one step.

from sklearn.feature_extraction.text import TfidfVectorizerimport pandas as pd
corpus = [    "the dog ran fast",    "the cat ran fast",    "the quantum physicist ran"]
# Create the vectorizervectorizer = TfidfVectorizer()
# Fit and transform the corpustfidf_matrix = vectorizer.fit_transform(corpus)
# Let's view the scores for the third documentfeature_names = vectorizer.get_feature_names_out()doc_3_scores = tfidf_matrix.toarray()[2]
df = pd.DataFrame({'Word': feature_names, 'TF-IDF': doc_3_scores})print(df[df['TF-IDF'] > 0].sort_values(by='TF-IDF', ascending=False))
# ->         Word    TF-IDF# -> 4  physicist  0.652491# -> 5    quantum  0.652491# -> 6        ran  0.385372# -> 7        the  0.385372

Notice that "physicist" and "quantum" get much higher scores than "ran" and "the", even though they all appeared exactly once in the third document.

Watch Out For

Watch Out For

It still ignores word order. Like Bag of Words, TF-IDF cannot understand phrasing, grammar, or negations. "Not good" and "good" will still result in identical vectors for the word "good".

Watch Out For

It struggles with synonyms. If Document A uses the word "car" and Document B uses the word "automobile", TF-IDF treats them as completely orthogonal concepts with zero similarity. It has no semantic understanding of the words, only statistical frequency. To capture actual meaning, you need dense embeddings like Word2Vec.

The Quick Version

  • Bag of Words overvalues common filler words. TF-IDF fixes this by applying a penalty to words that appear everywhere.
  • Term Frequency (TF) measures how often a word occurs in a specific document.
  • Inverse Document Frequency (IDF) measures how rare a word is across the entire corpus.
  • The final TF-IDF score is TF * IDF.
  • Words that are locally frequent but globally rare receive the highest scores, making it excellent for keyword extraction and traditional search engines.

Related concepts