Skip to content
AI360XpertBeta

GloVe

Count how often every word meets every other word across the whole corpus, then squeeze that giant table into short vectors that keep the ratios.

Counts for ice and steam against solid, gas, and water turn into ratios 68.9, 0.01, and 1.0, and vectors are trained so dot products match log counts
Counts for ice and steam against solid, gas, and water turn into ratios 68.9, 0.01, and 1.0, and vectors are trained so dot products match log counts

Why Does This Exist?

Early 2010s text had two camps. Count methods built one giant table over the whole corpus and factorised it. They saw global statistics but flopped on analogies like king minus man plus woman near queen. Window methods like word2vec slid five words at a time, predicted neighbours, and nailed analogies while never seeing corpus wide totals.

GloVe from 2014 keeps both. It starts from the global table, then trains vectors so their geometry encodes the ratios that carry meaning. This page covers that table to vectors path with the ice steam numbers. The window alternative lives in word2vec. The counting baseline lives in bag of words. The weighting trick that downplays common words lives in tf-idf.

Think of It Like This

Yearbook guest lists instead of party gossip

You want a map of who belongs with whom in a school. Gossip watches one party: Alice near Bob tonight, so they must be close. It misses that Alice and Charlie never share a party all year.

Guest lists do better. Collect every party list for the year, count how many times each pair co appears, then squeeze that giant attendance table into a small seating chart. Pairs that co appear far above chance sit together. Pairs that meet at chance rate sit apart, whether both popular or both absent.

Counts are the co-occurrence matrix, the seating chart is the embedding, and the squeeze that preserves ratios is the training. Gossip is the local window, guest lists are the global table.

Where it stops: seating charts place each student once. Real words shift meaning by sentence, and one fixed seat cannot be both river bank and money bank. Static vectors stop there by design.

How It Actually Works

Call XijX_{ij} the times word jj appears near word ii in a ten word window across the corpus. Rows are focus words, columns are context words, cells are counts. Xice,solid=1240X_{ice,solid} = 1240 means solid appeared near ice 1240 times.

Raw counts mislead because frequent words are large everywhere. Ratios fix it. Write P(k∣w)P(k \mid w) for the chance that kk appears near ww. The Stanford insight is that meaning lives in:

P(k∣ice)P(k∣steam)\frac{P(k \mid ice)}{P(k \mid steam)}

Worked example: ice, steam, and three probes

Use the diagram counts. Solid appears 1240 times near ice and 18 near steam. Gas appears 12 near ice and 1050 near steam. Water appears 8900 near ice and 9200 near steam.

Solid ratio:

1240/18≈68.91240 / 18 \approx 68.9

Large, because solid selects ice. Gas ratio:

12/1050≈0.01112 / 1050 \approx 0.011

Small, because gas selects steam. Water ratio:

8900/9200≈0.97≈1.08900 / 9200 \approx 0.97 \approx 1.0

Near one, because water goes with both and discriminates neither. A rare word near neither would also ratio near one. Only the ratio separates useful probes from noise, which raw counts cannot.

Training vectors to keep the ratios

GloVe forces dot products to match log counts. Call wiw_i and w~j\tilde{w}_j the two vectors for focus ii and context jj, with biases bib_i and b~j\tilde{b}_j. The objective pushes:

wi⊤w~j+bi+b~j≈log⁡Xijw_i^\top \tilde{w}_j + b_i + \tilde{b}_j \approx \log X_{ij}

log⁡\log means natural log. log⁡(1240)≈7.12\log(1240) \approx 7.12 and log⁡(18)≈2.89\log(18) \approx 2.89, so ice solid must dot about 4.23 higher than steam solid, while water pairs must dot almost equal. Fitting uses weighted least squares over nonzero cells, downweighting both ultra frequent pairs and one off noise, with vectors typically 100 to 300 wide. Unlike word2vec no fake prediction task runs. The table is the target directly.

Code

The point is the check: rebuild the three ratios and the log gap with plain arithmetic.

import math
ice_solid: float = 1240.0steam_solid: float = 18.0ice_gas: float = 12.0steam_gas: float = 1050.0ice_water: float = 8900.0steam_water: float = 9200.0
r_solid: float = ice_solid / steam_solidr_gas: float = ice_gas / steam_gasr_water: float = ice_water / steam_waterprint(f"{r_solid:.1f} {r_gas:.3f} {r_water:.2f}")# -> 68.9 0.011 0.97
gap: float = math.log(ice_solid) - math.log(steam_solid)print(f"{math.log(ice_solid):.2f} {math.log(steam_solid):.2f} {gap:.2f}")# -> 7.12 2.89 4.23

Solid ratios to 68.9, gas to 0.011, water to 0.97 near 1.0. Log counts give 7.12 and 2.89, gap 4.23, which is the dot product difference training must reproduce.

Watch Out For

Training from scratch on a small corpus

The table needs corpus wide totals to mean anything. On a few megabytes most cells are zero or one, ratios swing wildly, and vectors memorise noise.

Symptom: analogy tests flop while training loss looks fine, and nearest neighbours are spelling variants or rare typos. Fix: start from Stanford pretrained vectors trained on billions of tokens, or gather far more text before building your own table. Small data wants pretrained, not fresh.

Treating one vector as one meaning

GloVe assigns exactly one vector per spelling, so bank blends money and river uses into one point between them.

Symptom: similarity between bank and shore looks middling even in finance text, and analogies involving polysemy drift. Fix: use context aware models for sense sensitive work, and treat static cosine as topical relatedness rather than substitutability.

The Quick Version

  • GloVe builds XijX_{ij}, the global count of word jj near word ii, then trains vectors to reproduce it.
  • Meaning lives in ratios P(k∣ice)/P(k∣steam)P(k \mid ice) / P(k \mid steam), not raw counts.
  • Our counts give solid ratio 68.9, gas ratio 0.011, and water ratio 0.97 near 1.0.
  • Training forces wi⊤w~j+bi+b~j≈log⁡Xijw_i^\top \tilde{w}_j + b_i + \tilde{b}_j \approx \log X_{ij}, so ice solid must outdot steam solid by 4.23.
  • Vectors are 100 to 300 wide and static, so one spelling gets one point even with two senses.
  • Small corpora give noisy tables, so most teams start from pretrained vectors.