GloVe
Count how often every word meets every other word across the whole corpus, then squeeze that giant table into short vectors that keep the ratios.
Why Does This Exist?
Early 2010s text had two camps. Count methods built one giant table over the whole corpus and factorised it. They saw global statistics but flopped on analogies like king minus man plus woman near queen. Window methods like word2vec slid five words at a time, predicted neighbours, and nailed analogies while never seeing corpus wide totals.
GloVe from 2014 keeps both. It starts from the global table, then trains vectors so their geometry encodes the ratios that carry meaning. This page covers that table to vectors path with the ice steam numbers. The window alternative lives in word2vec. The counting baseline lives in bag of words. The weighting trick that downplays common words lives in tf-idf.
Think of It Like This
Yearbook guest lists instead of party gossip
You want a map of who belongs with whom in a school. Gossip watches one party: Alice near Bob tonight, so they must be close. It misses that Alice and Charlie never share a party all year.
Guest lists do better. Collect every party list for the year, count how many times each pair co appears, then squeeze that giant attendance table into a small seating chart. Pairs that co appear far above chance sit together. Pairs that meet at chance rate sit apart, whether both popular or both absent.
Counts are the co-occurrence matrix, the seating chart is the embedding, and the squeeze that preserves ratios is the training. Gossip is the local window, guest lists are the global table.
Where it stops: seating charts place each student once. Real words shift meaning by sentence, and one fixed seat cannot be both river bank and money bank. Static vectors stop there by design.
How It Actually Works
Call the times word appears near word in a ten word window across the corpus. Rows are focus words, columns are context words, cells are counts. means solid appeared near ice 1240 times.
Raw counts mislead because frequent words are large everywhere. Ratios fix it. Write for the chance that appears near . The Stanford insight is that meaning lives in:
Worked example: ice, steam, and three probes
Use the diagram counts. Solid appears 1240 times near ice and 18 near steam. Gas appears 12 near ice and 1050 near steam. Water appears 8900 near ice and 9200 near steam.
Solid ratio:
Large, because solid selects ice. Gas ratio:
Small, because gas selects steam. Water ratio:
Near one, because water goes with both and discriminates neither. A rare word near neither would also ratio near one. Only the ratio separates useful probes from noise, which raw counts cannot.
Training vectors to keep the ratios
GloVe forces dot products to match log counts. Call and the two vectors for focus and context , with biases and . The objective pushes:
means natural log. and , so ice solid must dot about 4.23 higher than steam solid, while water pairs must dot almost equal. Fitting uses weighted least squares over nonzero cells, downweighting both ultra frequent pairs and one off noise, with vectors typically 100 to 300 wide. Unlike word2vec no fake prediction task runs. The table is the target directly.
Code
The point is the check: rebuild the three ratios and the log gap with plain arithmetic.
import math
ice_solid: float = 1240.0steam_solid: float = 18.0ice_gas: float = 12.0steam_gas: float = 1050.0ice_water: float = 8900.0steam_water: float = 9200.0
r_solid: float = ice_solid / steam_solidr_gas: float = ice_gas / steam_gasr_water: float = ice_water / steam_waterprint(f"{r_solid:.1f} {r_gas:.3f} {r_water:.2f}")# -> 68.9 0.011 0.97
gap: float = math.log(ice_solid) - math.log(steam_solid)print(f"{math.log(ice_solid):.2f} {math.log(steam_solid):.2f} {gap:.2f}")# -> 7.12 2.89 4.23Solid ratios to 68.9, gas to 0.011, water to 0.97 near 1.0. Log counts give 7.12 and 2.89, gap 4.23, which is the dot product difference training must reproduce.
Watch Out For
Training from scratch on a small corpus
The table needs corpus wide totals to mean anything. On a few megabytes most cells are zero or one, ratios swing wildly, and vectors memorise noise.
Symptom: analogy tests flop while training loss looks fine, and nearest neighbours are spelling variants or rare typos. Fix: start from Stanford pretrained vectors trained on billions of tokens, or gather far more text before building your own table. Small data wants pretrained, not fresh.
Treating one vector as one meaning
GloVe assigns exactly one vector per spelling, so bank blends money and river uses into one point between them.
Symptom: similarity between bank and shore looks middling even in finance text, and analogies involving polysemy drift. Fix: use context aware models for sense sensitive work, and treat static cosine as topical relatedness rather than substitutability.
The Quick Version
- GloVe builds , the global count of word near word , then trains vectors to reproduce it.
- Meaning lives in ratios , not raw counts.
- Our counts give solid ratio 68.9, gas ratio 0.011, and water ratio 0.97 near 1.0.
- Training forces , so ice solid must outdot steam solid by 4.23.
- Vectors are 100 to 300 wide and static, so one spelling gets one point even with two senses.
- Small corpora give noisy tables, so most teams start from pretrained vectors.