Topic Modeling
Documents that share words share themes, so counting which words travel together reveals the hidden topics.
Why Does This Exist?
Ten thousand restaurant reviews land on your desk, completely unlabelled. Reading them all is out of the question, and supervised sorting is closed to you: text classification needs labelled examples and you have none. Still, someone wants to know what customers keep talking about.
Topic modeling answers from the raw text alone. It watches which words show up in the same documents and groups the co-travellers, so out come themes like service, food quality and price without a single label. Start from bag of words for the counting behind it, and compare notes with k-means, its cousin for clustering rows instead of words.
Think of It Like This
The recycling sorting line
A recycling plant receives one mixed stream: glass, aluminium and paper tumbling together, none of it labelled. Nobody reads each item. The line shakes the stream and watches which clues travel together: heft plus a clink means glass, lightness plus a crinkle means cans. Items that share clues share a chute.
Topic modeling sorts documents the same way, with words as the clues. Bank, loan and interest keep arriving in the same documents, so they share a chute called finance.
Where it stops: glass is glass, a fixed truth the plant can check. Topics are statistical conveniences with no ground truth, so rerun the model and the chutes shuffle. Nobody at the plant renames glass every morning.
How It Actually Works
The classic algorithm is Latent Dirichlet Allocation, LDA. It imagines documents are generated by a recipe with two steps: each document picks a mix of topics, then each topic picks the actual words. You observe only the words, so the algorithm works backwards, hunting for the topic mixes and word mixes that make the observed documents most likely.
Its output is two tables. The document-topic table says each document is, say, 80 percent politics and 20 percent finance. The topic-word table defines each topic as probabilities over words, like . Modern variants such as BERTopic keep the same goal but cluster contextual embeddings instead of raw counts, which separates money-bank from river-bank.
Worked example
Vocabulary of four words: bank, loan, river, stream. Count them in two piles of documents. The finance pile holds bank 4 times, loan 3 times, stream once, river never: . Divide by the total of 8 and the topic is a proper distribution:
The nature pile holds , which normalises to . Now a new review arrives with bank 3 times, loan twice and river once: . Five of its six content words come from the finance mix, so it lands about in finance. That is the number the diagram draws.
Code
The point is the normalisation: raw counts become probabilities by dividing by the pile total, and a document's topic share is the fraction of its words each topic explains.
topic_finance = [4, 3, 0, 1] # bank, loan, river, streamtopic_nature = [0, 1, 4, 3]
def dist(counts): total = sum(counts) return [c / total for c in counts]
print([round(p, 3) for p in dist(topic_finance)]) # -> [0.5, 0.375, 0.0, 0.125]print([round(p, 3) for p in dist(topic_nature)]) # -> [0.0, 0.125, 0.5, 0.375]
doc = [3, 2, 1, 0] # bank x3, loan x2, river x1finance_words = doc[0] + doc[1]print(finance_words, sum(doc), round(finance_words / sum(doc), 3)) # -> 5 6 0.833Five of six words, 0.833 finance. Same arithmetic as the worked example, now executable.
Watch Out For
A topic made of glue words
One topic comes back as the, and, of, is, to. That is not a theme, it is the most common words in the language bunching together, and it drowns the real topics. The symptom is unmistakable: a topic whose top words could belong to any document ever written.
Strip stop words and the most frequent filler before fitting, and consider dropping words that appear in nearly every document. Refit and check that every topic's top words now belong together. Preprocessing is half the method here.
Picking the topic count blind
You must hand LDA its topic count up front, and there is no correct answer waiting. Too small and politics merges with sport into one useless bucket. Too large and finance shatters into bank, loan and interest fragments that all mean the same thing.
Fit several values of and read the top words of each, like tasting several grinds before buying the bag. Coherence scores help rank candidates, but the final judge is whether a human can name each topic in three words.
The Quick Version
- Topic modeling groups co-occurring words into themes across unlabelled documents, with no labels needed.
- LDA assumes each document mixes topics and each topic mixes words, then works backwards from the observed counts.
- Counts become probabilities by dividing by the pile total, and a document's share follows its words.
- A review with five finance words out of six lands 0.833 in finance.
- Strip stop words first or a glue-word topic eats the signal.
- Fit several values of , because too few topics merge and too many shatter.