Word2Vec
Word2Vec turns each word into a short list of numbers, learned by guessing neighbours, so words that share company land near each other.
Why Does This Exist?
Counting words gives you tallies with no sense of meaning. Bag of words gives every spelling its own axis, so cat sits as far from dog as from skyscraper. With fifty thousand words in the list, each sentence becomes a row that is almost all zeros, and synonyms stay strangers.
TF-IDF fixes ranking by downweighting filler, yet car and automobile still live on different axes. Search misses paraphrase. Clustering splits one topic by wording. Any model reading those rows must relearn each synonym pair from labels.
Word2Vec made dense vectors the standard fix. Learn one short vector per word from neighbours, then compare with a ruler. Words that keep the same company land close together, even when spellings share nothing. You will feel this the first time cat returns dog instead of a random noun.
This page covers that neighbour trick and its two training setups. Counting lives in bag of words. Rarity weighting lives in TF-IDF. Vectors for whole sentences come later in sentence embeddings.
You need one idea first. A one-hot vector gives each word its own axis. With vocabulary size , each word is a long row with a single 1 and the rest 0. Any two different words share nothing, so every pair sits equally far. That equal distance pain is what dense vectors remove.
Think of It Like This
Shoppers mapped by the stores they visit
You never ask shoppers what they like. You watch where they shop. Two shoppers who visit the same bakery, the same bookshop, and the same park stall get pins placed close together on your map. Shoppers with no stores in common land far apart.
Training does the placing. Each visit nudges pins, so shared habits pull pins together while different habits hold them apart. After many visits the map groups bakers with bakers, without anyone naming the groups first.
Words are shoppers, neighbouring words are stores, and pin coordinates are vectors. Cat and dog share neighbours like fed, pet, and chased, so their pins end up side by side.
Where it stops: shoppers pick stores for price and distance as well as taste, and your map is flat while word vectors use a hundred or more directions. The map explains neighbour pressure. Only the arithmetic explains why a hundred directions beat two.
How It Actually Works
Give each word two short vectors, one for when it plays center and one for when it plays neighbour. Call the vocabulary size , meaning the count of distinct words. Call the embedding width , meaning numbers per vector. The model keeps a by table, so each row is one word vector. Training moves rows, then keeps the table and drops the rest.
Skip-gram predicts outward
Pick a center word , meaning the word in the middle of the window. Look at one neighbour , meaning a word beside it. Score every vocabulary word by dot product with , then turn scores into probabilities with softmax. The update pushes mass toward the true neighbours and away from the rest.
In symbols, with for the center vector and for neighbour vectors:
Here is the center vector, meaning the row for . Here is the neighbour vector, meaning the row for . The dot is similarity, meaning large when directions agree. The makes scores positive. The sum over runs over all words, so the probabilities total 1. The softmax name is that normalising step.
CBOW averages inward
CBOW flips the direction. Average the neighbour vectors into one context vector , meaning the mean of the surrounding words. Predict the center from that average with the same softmax. Frequent words train fast here because each window gives one center update, while skip-gram pays one update per neighbour and spends more time on rare words.
Worked example: scores to probabilities
Take center fox with four candidates. Raw scores are 2, 1, 0, and minus 1. Exponentiate each:
Add them:
Divide each by the total:
The top candidate owns almost two thirds of the mass, yet the rest keep small shares that still move their vectors. Those four numbers sit on the left of the diagram as bars, longest on top.
Worked example: closeness you can quote
Fix the ruler first:
Here and are two word vectors, meaning two short rows. Here is the dot product, meaning matching entries multiplied then added. Each is length, meaning square root of sum of squares. Dividing by both lengths keeps scores between minus 1 and 1.
Take toy vectors for cat and for dog. Their dot product is , from . Their lengths are about and . So:
Against for car, the dot with is and the cosine is about . Same ruler, gap from near identical to distant. That gap is what search ranks on, and the ruler is cosine similarity. The right of the diagram draws these three points at those exact scores.
Code
The point is the check: rebuild the four probabilities and both cosines from the worked examples.
import math
def softmax(scores: list[float]) -> list[float]: m: float = max(scores) # shift keeps exp safe, leaves ratios unchanged exps: list[float] = [math.exp(s - m) for s in scores] tot: float = sum(exps) return [e / tot for e in exps]
def cosine(u: list[float], v: list[float]) -> float: dot: float = u[0] * v[0] + u[1] * v[1] nu: float = math.sqrt(u[0] ** 2 + u[1] ** 2) nv: float = math.sqrt(v[0] ** 2 + v[1] ** 2) return dot / (nu * nv)
print([f"{p:.3f}" for p in softmax([2.0, 1.0, 0.0, -1.0])]) # -> ['0.644', '0.237', '0.087', '0.032']print(f"{cosine([0.9, 0.3], [0.8, 0.4]):.2f}") # -> 0.99print(f"{cosine([0.9, 0.3], [0.1, 0.9]):.2f}") # -> 0.42The printout matches the hand arithmetic. Change fox scores to equal values and watch all four probabilities flatten to 0.25, which is why uniform scores teach nothing.
Watch Out For
Asking bank for one meaning
Word2Vec keeps one row per spelling. River bank and money bank pull that single row toward both neighbourhoods, so the final point lands between them and near neither. The symptom is concrete: neighbours of bank mix river and money terms, and similarity to each sense sits near 0.5 instead of near 0.9.
Fix: do not ship single sense vectors where polysemy decides the label. Either keep the window task for first baselines and move to contextual vectors for sense work, or split the spelling by usage before training. One address cannot house two meanings.
Shipping a lookup that crashes on new words
A word outside training has no row. The symptom is a KeyError at serve time on the first typo, new name, or fresh slang, exactly when traffic peaks. Teams patch by skipping unknown words, which silently shortens inputs and shifts every window.
Fix: decide the fallback before launch. Map rare spellings to subword pieces, keep an explicit unknown vector trained with dropout, or route unknowns to sentence embeddings that handle pieces. Log the unknown rate daily, because a rising rate means the vocabulary is stale.
The Quick Version
- Word2Vec learns one short vector per word by predicting neighbours, not by counting tallies.
- Skip-gram predicts outward from one word and suits rare words, while CBOW averages inward and trains faster.
- Scores 2, 1, 0 and minus 1 become 0.644, 0.237, 0.087 and 0.032, so the top neighbour pulls hardest.
- Toy vectors put cat near dog at 0.99 and cat near car at 0.42 on the same cosine ruler.
- One vector per spelling cannot hold two senses, so bank lands between river and money.
- New words have no row, so plan a fallback instead of crashing at serve time.