Skip to content
AI360XpertBeta

Word2Vec

Word2Vec turns each word into a short list of numbers, learned by guessing neighbours, so words that share company land near each other.

Skip-gram turns scores 2, 1, 0 and minus 1 into probabilities 0.644 to 0.032, pulling cat near dog and leaving car far
Skip-gram turns scores 2, 1, 0 and minus 1 into probabilities 0.644 to 0.032, pulling cat near dog and leaving car far

Why Does This Exist?

Counting words gives you tallies with no sense of meaning. Bag of words gives every spelling its own axis, so cat sits as far from dog as from skyscraper. With fifty thousand words in the list, each sentence becomes a row that is almost all zeros, and synonyms stay strangers.

TF-IDF fixes ranking by downweighting filler, yet car and automobile still live on different axes. Search misses paraphrase. Clustering splits one topic by wording. Any model reading those rows must relearn each synonym pair from labels.

Word2Vec made dense vectors the standard fix. Learn one short vector per word from neighbours, then compare with a ruler. Words that keep the same company land close together, even when spellings share nothing. You will feel this the first time cat returns dog instead of a random noun.

This page covers that neighbour trick and its two training setups. Counting lives in bag of words. Rarity weighting lives in TF-IDF. Vectors for whole sentences come later in sentence embeddings.

You need one idea first. A one-hot vector gives each word its own axis. With vocabulary size VV, each word is a VV long row with a single 1 and the rest 0. Any two different words share nothing, so every pair sits equally far. That equal distance pain is what dense vectors remove.

Think of It Like This

Shoppers mapped by the stores they visit

You never ask shoppers what they like. You watch where they shop. Two shoppers who visit the same bakery, the same bookshop, and the same park stall get pins placed close together on your map. Shoppers with no stores in common land far apart.

Training does the placing. Each visit nudges pins, so shared habits pull pins together while different habits hold them apart. After many visits the map groups bakers with bakers, without anyone naming the groups first.

Words are shoppers, neighbouring words are stores, and pin coordinates are vectors. Cat and dog share neighbours like fed, pet, and chased, so their pins end up side by side.

Where it stops: shoppers pick stores for price and distance as well as taste, and your map is flat while word vectors use a hundred or more directions. The map explains neighbour pressure. Only the arithmetic explains why a hundred directions beat two.

How It Actually Works

Give each word two short vectors, one for when it plays center and one for when it plays neighbour. Call the vocabulary size VV, meaning the count of distinct words. Call the embedding width dd, meaning numbers per vector. The model keeps a VV by dd table, so each row is one word vector. Training moves rows, then keeps the table and drops the rest.

Skip-gram predicts outward

Pick a center word ww, meaning the word in the middle of the window. Look at one neighbour cc, meaning a word beside it. Score every vocabulary word by dot product with ww, then turn scores into probabilities with softmax. The update pushes mass toward the true neighbours and away from the rest.

In symbols, with vv for the center vector and uu for neighbour vectors:

p(c∣w)=exp⁡(uc⋅vw)∑j=1Vexp⁡(uj⋅vw)p(c \mid w) = \frac{\exp(u_c \cdot v_w)}{\sum_{j=1}^{V} \exp(u_j \cdot v_w)}

Here vwv_w is the center vector, meaning the row for ww. Here ucu_c is the neighbour vector, meaning the row for cc. The dot uc⋅vwu_c \cdot v_w is similarity, meaning large when directions agree. The exp⁡\exp makes scores positive. The sum over jj runs over all VV words, so the VV probabilities total 1. The softmax name is that normalising step.

CBOW averages inward

CBOW flips the direction. Average the neighbour vectors into one context vector vˉ\bar{v}, meaning the mean of the surrounding words. Predict the center from that average with the same softmax. Frequent words train fast here because each window gives one center update, while skip-gram pays one update per neighbour and spends more time on rare words.

Worked example: scores to probabilities

Take center fox with four candidates. Raw scores are 2, 1, 0, and minus 1. Exponentiate each:

exp⁡([2, 1, 0, −1])=[7.389, 2.718, 1.000, 0.368]\exp([2,\ 1,\ 0,\ -1]) = [7.389,\ 2.718,\ 1.000,\ 0.368]

Add them:

7.389+2.718+1.000+0.368=11.4757.389 + 2.718 + 1.000 + 0.368 = 11.475

Divide each by the total:

[7.389, 2.718, 1.000, 0.368]/11.475=[0.644, 0.237, 0.087, 0.032][7.389,\ 2.718,\ 1.000,\ 0.368] / 11.475 = [0.644,\ 0.237,\ 0.087,\ 0.032]

The top candidate owns almost two thirds of the mass, yet the rest keep small shares that still move their vectors. Those four numbers sit on the left of the diagram as bars, longest on top.

Worked example: closeness you can quote

Fix the ruler first:

cos⁡(u,v)=u⋅v∥u∥ ∥v∥\cos(u, v) = \frac{u \cdot v}{\lVert u \rVert \, \lVert v \rVert}

Here uu and vv are two word vectors, meaning two short rows. Here u⋅vu \cdot v is the dot product, meaning matching entries multiplied then added. Each ∥⋅∥\lVert \cdot \rVert is length, meaning square root of sum of squares. Dividing by both lengths keeps scores between minus 1 and 1.

Take toy vectors a=[0.9, 0.3]a = [0.9,\ 0.3] for cat and b=[0.8, 0.4]b = [0.8,\ 0.4] for dog. Their dot product is 0.840.84, from 0.72+0.120.72 + 0.12. Their lengths are about 0.9490.949 and 0.8940.894. So:

0.84/(0.949×0.894)≈0.990.84 / (0.949 \times 0.894) \approx 0.99

Against c=[0.1, 0.9]c = [0.1,\ 0.9] for car, the dot with aa is 0.360.36 and the cosine is about 0.420.42. Same ruler, gap from near identical to distant. That gap is what search ranks on, and the ruler is cosine similarity. The right of the diagram draws these three points at those exact scores.

Code

The point is the check: rebuild the four probabilities and both cosines from the worked examples.

import math

def softmax(scores: list[float]) -> list[float]:    m: float = max(scores)  # shift keeps exp safe, leaves ratios unchanged    exps: list[float] = [math.exp(s - m) for s in scores]    tot: float = sum(exps)    return [e / tot for e in exps]

def cosine(u: list[float], v: list[float]) -> float:    dot: float = u[0] * v[0] + u[1] * v[1]    nu: float = math.sqrt(u[0] ** 2 + u[1] ** 2)    nv: float = math.sqrt(v[0] ** 2 + v[1] ** 2)    return dot / (nu * nv)

print([f"{p:.3f}" for p in softmax([2.0, 1.0, 0.0, -1.0])])  # -> ['0.644', '0.237', '0.087', '0.032']print(f"{cosine([0.9, 0.3], [0.8, 0.4]):.2f}")  # -> 0.99print(f"{cosine([0.9, 0.3], [0.1, 0.9]):.2f}")  # -> 0.42

The printout matches the hand arithmetic. Change fox scores to equal values and watch all four probabilities flatten to 0.25, which is why uniform scores teach nothing.

Watch Out For

Asking bank for one meaning

Word2Vec keeps one row per spelling. River bank and money bank pull that single row toward both neighbourhoods, so the final point lands between them and near neither. The symptom is concrete: neighbours of bank mix river and money terms, and similarity to each sense sits near 0.5 instead of near 0.9.

Fix: do not ship single sense vectors where polysemy decides the label. Either keep the window task for first baselines and move to contextual vectors for sense work, or split the spelling by usage before training. One address cannot house two meanings.

Shipping a lookup that crashes on new words

A word outside training has no row. The symptom is a KeyError at serve time on the first typo, new name, or fresh slang, exactly when traffic peaks. Teams patch by skipping unknown words, which silently shortens inputs and shifts every window.

Fix: decide the fallback before launch. Map rare spellings to subword pieces, keep an explicit unknown vector trained with dropout, or route unknowns to sentence embeddings that handle pieces. Log the unknown rate daily, because a rising rate means the vocabulary is stale.

The Quick Version

  • Word2Vec learns one short vector per word by predicting neighbours, not by counting tallies.
  • Skip-gram predicts outward from one word and suits rare words, while CBOW averages inward and trains faster.
  • Scores 2, 1, 0 and minus 1 become 0.644, 0.237, 0.087 and 0.032, so the top neighbour pulls hardest.
  • Toy vectors put cat near dog at 0.99 and cat near car at 0.42 on the same cosine ruler.
  • One vector per spelling cannot hold two senses, so bank lands between river and money.
  • New words have no row, so plan a fallback instead of crashing at serve time.