Topic Modeling

Why Does This Exist?

Imagine you are given 10,000 customer reviews of a restaurant.

You don't have time to read them, and they are completely unlabelled, so you can't train a Text Classifier to sort them into buckets. You just want to know: What are the main things people are talking about?

Topic modeling is an unsupervised learning technique that solves this. It scans the raw, unlabelled text and automatically groups words into clusters (topics), revealing that 40% of the reviews discuss "wait time and service", 35% discuss "food quality", and 25% discuss "price and value".

Think of It Like This

Think of It Like This

Imagine you are given a bowl of mixed M&Ms, Skittles, and Reese's Pieces. They are all jumbled together.

You don't know the names of the candies, but you notice that the M&Ms always have an 'M' stamped on them, the Skittles have an 'S', and the Reese's taste like peanut butter. Based purely on observing these co-occurring features, you can separate the bowl into three distinct piles.

Topic modeling does this with words. If it constantly sees the words "bank", "interest", and "loan" appearing in the same documents, it groups them into a pile and says "this is Topic 1" (Finance).

How It Actually Works

For over a decade, the gold standard algorithm for topic modeling was Latent Dirichlet Allocation (LDA).

Latent Dirichlet Allocation (LDA)

LDA is a generative probabilistic model. It operates on two core assumptions:

1. Every document is a mixture of topics. (e.g., A news article might be 80% Politics and 20% Finance).
2. Every topic is a mixture of words. (e.g., The "Politics" topic has a high probability of generating words like "election", "senate", and "vote").

When you feed 10,000 documents into LDA, you tell it how many topics you want it to find (e.g., $K=5$). The algorithm works backwards. It iterates through the documents, constantly adjusting the probabilities of which words belong to which topics, and which topics belong to which documents, until it finds the optimal mathematical arrangement.

The output is two matrices:

• Document-Topic Matrix: Tells you the topic breakdown of every document.
• Topic-Word Matrix: Gives you the top keywords that define each of the 5 topics.

The Modern Approach: BERTopic

While LDA relies on simple word counts (Bag of Words), modern topic modeling uses Transformer embeddings. Algorithms like BERTopic first convert every document into a dense Sentence Embedding. It then applies dimensionality reduction (UMAP) and clustering (HDBSCAN) to group the documents in vector space. Because it uses contextual embeddings, it understands that "bank" (money) and "bank" (river) belong in completely different topics—something LDA struggles with.

Watch Out For

Watch Out For

Topics are unnamed. LDA will return a topic defined by the words: [orbit, space, launch, moon, nasa]. It is entirely up to you, the human, to look at that list of words and assign the label "Space Exploration". If you run a topic model and it returns [the, is, and, of], you forgot to remove your stop words during preprocessing.

Watch Out For

Choosing K is difficult. Because it is unsupervised, there is no "correct" number of topics ($K$). If you set $K$ too low, distinct topics merge into uselessly broad buckets. If you set $K$ too high, a single coherent topic splits into redundant fragments. You must rely on coherence metrics and human judgment to find the right $K$.

The Quick Version

• Topic modeling automatically discovers hidden themes in large datasets of unlabelled text.
• It is an unsupervised learning technique, unlike text classification.
• LDA is the classical probabilistic algorithm that assumes documents are mixtures of topics, and topics are mixtures of words.
• Modern variants like BERTopic use transformer embeddings and clustering to achieve context-aware topic extraction.

What to Read Next

• Text Classification
• Sentence Embeddings