Skip to content
AI360Xpert
Core ML

Bag of Words

A simple text representation that counts how many times each word appears, ignoring grammar, word order, and context.

Bag of Words discards all syntax and order, representing a document simply as a vector of its word frequencies.
Bag of Words discards all syntax and order, representing a document simply as a vector of its word frequencies.

Why Does This Exist?

Machine learning algorithms require numerical inputs, usually fixed-size vectors. Text is messy, variable in length, and categorical. We need a way to mathematically represent a sentence so that a model can compare two sentences and determine if they are similar.

Bag of Words (BoW) is the simplest, most intuitive way to do this. It assumes that the meaning of a document is entirely captured by the words it contains and their frequencies, regardless of the order they appear in.

Think of It Like This

Think of It Like This

Imagine you take a page of a book, cut out every single word with scissors, and throw them all into a bag. You shake the bag up.

You have lost all the grammar, the sentence structure, and the story narrative. But if you reach in and pull out the words "goal", "referee", "ball", and "score" over and over again, you can still confidently guess that the page was about soccer. Bag of Words bets that word counts alone are enough to classify the topic of a document.

How It Actually Works

Creating a Bag of Words representation involves two steps: defining a vocabulary, and counting.

1. The Vocabulary

First, you gather your entire training dataset (your corpus) and extract every unique token (usually after text preprocessing). You assign a fixed index to every unique word.

If your corpus only consists of the sentences "the dog ran" and "the cat ran", your vocabulary might be: [the, dog, ran, cat]. The size of this vocabulary (4) determines the length of your output vectors.

2. The Vectorization

To represent a new document, you create an array of zeros equal to the size of your vocabulary. You iterate through the document, and for every word, you increment the integer at that word's corresponding index in the array.

Given the vocabulary [the, dog, ran, cat], the sentence "the dog ran" becomes: [1, 1, 1, 0].

A sentence with repeated words, like "the dog dog", becomes: [1, 2, 0, 0].

Sparsity

In the real world, a vocabulary might contain 50,000 unique words. A single tweet might contain only 15 words. Therefore, the BoW vector for that tweet will be an array of length 50,000 containing 49,985 zeros. We call this a sparse vector. In practice, these are stored using compressed sparse matrix formats to save memory.

Show Me the Code

In Python, scikit-learn provides the CountVectorizer, which automatically builds the vocabulary and generates the sparse Bag of Words matrix.

from sklearn.feature_extraction.text import CountVectorizer
corpus = [    "the dog ran",    "the cat ran",    "the dog and the dog ran"]
# Create the vectorizervectorizer = CountVectorizer()
# Fit the vocabulary and transform the corpus into BoW vectorsbow_matrix = vectorizer.fit_transform(corpus)
# The learned vocabulary maps words to indicesprint("Vocabulary:", vectorizer.vocabulary_)# -> {'the': 3, 'dog': 2, 'ran': 4, 'cat': 1, 'and': 0}
# The resulting vectorsprint("Dense matrix:\n", bow_matrix.toarray())# -> [[0 0 1 1 1]#     [0 1 0 1 1]#     [1 0 2 2 1]]

Watch Out For

Watch Out For

Loss of context and word order. Bag of Words thinks "The dog bit the man" and "The man bit the dog" are exactly identical, because they have the exact same word counts. It cannot capture syntax, sarcasm, or negation (e.g., "not good" vs "good").

Watch Out For

Vocabulary explosion and OOV words. If you train on Wikipedia, your vocabulary might be millions of words long, causing massive memory consumption. Furthermore, if the model encounters a word in production that was not in the training vocabulary (an Out Of Vocabulary word), Bag of Words completely ignores it.

The Quick Version

  • Bag of Words converts text into fixed-length numerical vectors based on word counts.
  • It first defines a vocabulary of all known words, assigning each a fixed column index.
  • A document's vector contains the frequency of each vocabulary word present in that document.
  • It completely discards word order and grammar.
  • The resulting vectors are highly sparse (mostly zeros) because any given document only uses a tiny fraction of the total vocabulary.

Related concepts