Skip to content
AI360XpertBeta
Core ML

Visual explainer

Word2Vec

Context prediction learns word vectors. Follow an explicit window, two vector tables, a four-word softmax, CBOW, and negative sampling.

Distinct one-hot word vectors have equal pairwise distances.
Distinct one-hot word vectors have equal pairwise distances.

Word2vec learns word vectors by predicting words from local text contexts. A one-hot ID alone supplies no similarity ranking: in a three-word illustration, fox [1,0,0], jumps [0,1,0] and moon [0,0,1] all have pairwise distance √2 and pairwise cosine zero. The equal bars measure those distances, with 100 SVG units per distance unit. The arrows between names identify pairs, not prediction directions.

Skip-gram uses a center word to predict context words. Continuous bag-of-words, or CBOW, reverses that direction by using surrounding words to predict a center. Both learn representations through a prediction objective rather than placing semantically named points by hand.

Make Local Training Pairs

A radius-two window creates four fox-to-context training pairs.
A radius-two window creates four fox-to-context training pairs.

In quick brown fox jumps high, a fixed radius of two gives fox four context targets: quick, brown, jumps and high. Each position offset is relative to fox; the center itself is excluded. The resulting skip-gram pairs are fox→quick, fox→brown, fox→jumps and fox→high. Nearby words need not be synonyms.

This follows the skip-gram context objective, equation 1. The fixed window is illustrative; training can sample a smaller radius and subsample frequent words. Neither a word's distance in the sentence nor the diagram's chip spacing is a learned vector distance.

Score Across Two Tables

Skip-gram scores a center–context pair using two vector tables.
Skip-gram scores a center–context pair using two vector tables.

For full softmax and negative sampling, maintain input vectors v for center roles and output vectors u for context roles. A score is s=u(wo)⋅v(wi)s=u(w_o)\cdot v(w_i). In our separate four-word toy vocabulary, v(fox)=[1,0] and the output rows are u(jumps)=[2,0], u(brown)=[1,1], u(fox)=[0,1] and u(moon)=[−1,0]. These chosen values are not trained results, and this toy scoring vocabulary differs from the window example.

The drawing isolates fox→jumps. Both vectors share an origin at 100 SVG units per coordinate: the input ring is one unit long, the output diamond two. Because the input vector is unit length and both directions align, the dot product is 2. Scores for the four output rows are [2,1,0,−1]. The same word can have different input and output rows, and a training score is not cosine similarity between stored input embeddings.

Predict Inward With CBOW

CBOW averages context vectors to predict the center word.
CBOW averages context vectors to predict the center word.

This scene uses a deliberately separate CBOW fixture: quick [1,3], brown [3,1], jumps [2,2] and high [2,2]. At 60 SVG units per coordinate, quick and brown sit at opposite ends of the dashed segment. The jumps and high points coincide. The mint ring marks the mean, also at that same point:

h=([1,3]+[3,1]+[2,2]+[2,2])/4=[2,2].h=([1,3]+[3,1]+[2,2]+[2,2])/4=[2,2].

An output classifier uses h to predict the center, whose training target is fox. The arrow identifies that task; it does not claim a measured prediction probability. Averaging drops context order. The original architecture paper, section 3.1, describes the shared projection and this averaging, contrasting it with skip-gram's outward prediction.

Compare Stored Directions

Cosine ranks orange above car in the declared toy embedding table.
Cosine ranks orange above car in the declared toy embedding table.

Switch explicitly to an independent illustrative table of stored word vectors: apple [1,0], orange [3,1] and car [1,3]. The rays use 60 SVG units per coordinate. Orange's angular separation from apple is about 18.435°, versus about 71.565° for car. Cosine scores are 3/10≈.9486833/\sqrt{10}\approx.948683 and 1/10≈.3162281/\sqrt{10}\approx.316228.

This explains how one might rank neighbors after training, without claiming these points came from our earlier update or a measured Word2Vec model. Learned axes need not have human-readable meanings; real cosines may be negative. Choose which table or combination to export deliberately. The similarity ranking differs from the context-prediction likelihood.

Sample A Binary Objective

Negative sampling uses independent data-versus-noise sigmoid terms.
Negative sampling uses independent data-versus-noise sigmoid terms.

Return to the fox input and output table from the scoring scene. Take observed pair fox→jumps and two chosen noise draws, brown and moon. Label jumps 1 for data and both draws 0 for noise. With scores 2, 1 and −1, the independent sigmoid scores σ(s)=1/(1+e−s)\sigma(s)=1/(1+e^{-s}) are .880797, .731059 and .268941. Each ruler has its own unit interval, at 200 SVG units per probability unit; they do not form a vocabulary distribution.

The minimized loss for these particular draws is

L=−log⁡σ(2)−log⁡σ(−1)−log⁡σ(1)≈1.753451.L=-\log\sigma(2)-\log\sigma(-1)-\log\sigma(1)\approx1.753451.

Brown currently receives a high data score despite its noise label, so that term pushes its score downward. A sampled noise word can occur in genuine contexts too; it is not a known semantic opposite. Negative sampling, equation 4, changes the objective to sampled binary terms. It avoids scoring every vocabulary row for each pair, but its samples and their number affect the learning signal.

Normalize The Full Vocabulary

Full softmax normalizes all four context scores together.
Full softmax normalizes all four context scores together.

Compare a different objective on exactly the same original scores [2,1,0,−1]. Full softmax computes pj=esj/Zp_j=e^{s_j}/Z, where Z=e2+e1+e0+e−1≈11.475217Z=e^2+e^1+e^0+e^{-1}\approx11.475217. The probabilities for jumps, brown, fox and moon are approximately [.643914,.236883,.087144,.032059]. The four segment widths partition a single 400-unit strip and sum to one. Leaders identify the small fox and moon segments without squeezing labels inside them.

For observed target jumps, the negative log-likelihood is −log⁡(.643914)≈.440190-\log(.643914)\approx.440190. Fox remains a possible vocabulary output even though the current window does not supply fox as its own target. Full normalization scores every output word in this fixture; negative sampling uses only its observed and sampled rows. Their losses are different objectives, so comparing their numerical magnitudes does not rank their quality.

Move A Learned Vector

A negative-sampling update moves the center vector using the observed and noise pairs.
A negative-sampling update moves the center vector using the observed and noise pairs.

Use the original negative-sampling state, not a state updated by softmax. For one pair with binary label y, the derivative with respect to its dot score is σ(s)−y\sigma(s)-y. Summing the observed jumps term and the brown/moon noise terms gives the center-vector gradient

∇vL=(σ(2)−1)[2,0]+σ(1)[1,1]+σ(−1)[−1,0]≈[.223711,.731059].\nabla_vL=(\sigma(2)-1)[2,0]+\sigma(1)[1,1]+\sigma(-1)[-1,0] \approx[.223711,.731059].

At chosen learning rate .1, vnew=[1,0]−.1∇vL≈[.977629,−.073106]v_{new}=[1,0]-.1\nabla_vL\approx[.977629,-.073106]. The circle is the old vector and the diamond the new one. This is a cropped local view at 1,400 SVG units per coordinate, with the axes crossing at old value (1,0), not at the global origin. The displacement is about (−31.320,+102.348) screen units because SVG y increases downward. The drawn movement therefore matches the computed parameter change.

Both tables learn. A simultaneous step evaluated entirely at the old state also gives u(jumps)≈[2.011920,0], u(brown)≈[.926894,1] and u(moon)≈[−1.026894,0]. The unsampled output fox row stays unchanged in this step. Only the input-vector movement is pictured. The rate and numbers are teaching settings, not recommended hyperparameters or an empirical performance result.

Change The Evidence Window

Shrinking this window from radius two to one removes the two outer training pairs.
Shrinking this window from radius two to one removes the two outer training pairs.

Hold the same five tokens and center fixed. Radius two supplies four targets, while radius one keeps only brown and jumps. Dashed outer chips and crosses mark quick and high as excluded from this smaller window. The brackets represent included positions, with fox still excluded as its own target.

This changes the training evidence as well as the amount of work. The original architecture paper samples window sizes; the later skip-gram paper also describes subsampling frequent words. The drawing isolates radius alone rather than simulating those stochastic procedures. Evaluate window and sampling choices on the actual corpus and task rather than using a universal corpus-size rule.

Where It Breaks

Different bank contexts still select the same Word2Vec input row.
Different bank contexts still select the same Word2Vec input row.

The two bank occurrences select the same illustrative input row [.4,1.2] despite different river and loan contexts. The two lookup paths terminate in exactly one stored row. More training can change that row's mixed usage information, but it cannot make a word-type lookup depend on this occurrence's surrounding tokens.

This does not force bank to a geometric midpoint or imply that a downstream context-aware model cannot distinguish its senses. Use surrounding-token computation when the current sense matters. Word2vec supplies static word representations; it does not by itself specify an occurrence-sensitive sentence encoder. Its context learning also does not guarantee useful semantic neighbors for every corpus or task.

The Quick Version

  • Distinct one-hot words have equal pairwise distances.
  • Skip-gram turns local windows into center–context training pairs.
  • Input and output tables supply dot-product prediction scores.
  • CBOW averages contexts to predict a center word.
  • Cosine ranking differs from context-prediction likelihood.
  • Negative sampling uses independent binary terms.
  • Full softmax normalizes scores over the entire vocabulary.
  • Observed and noise pairs jointly move learned vector coordinates.
  • Window radius changes which pairs supply training evidence.
  • Static Word2Vec rows cannot adapt to the current word sense.