Skip to content
AI360XpertBeta
Core ML

Visual explainer

Word Embeddings

Word embeddings map word identities to learned vectors. Follow a toy lookup, compare cosine and distance, and see the limit of one vector per word.

One-hot word identities have equal pairwise distances.
One-hot word identities have equal pairwise distances.

Word embeddings map word identities to learned numerical vectors. A static word table supplies the same row whenever the same ID is looked up. A one-hot vector only identifies that row: cat [1,0,0], dog [0,1,0] and car [0,0,1] all have pairwise dot product zero and Euclidean distance √2. The two equal-length bars compare dog and car with cat; their identities alone supply no semantic ranking.

Select A Learned Row

An embedding lookup selects a word's learned row of numbers.
An embedding lookup selects a word's learned row of numbers.

Our illustrative table E has cat [2,1], dog [2,2] and car [0,3]. Cat's ID selects the outlined first row, yielding [2,1]. Multiplying [1,0,0] by E gives the same result, but implementations can index the table directly without materializing a one-hot vector. The PyTorch Embedding documentation describes this lookup behavior.

The coordinates are chosen teaching values, not measured embeddings. Dense means a compact distributed representation here; it does not require every entry to be nonzero. Learning adjusts the table through an objective rather than assigning named English properties to its dimensions.

Compare Directions

Cat points closer in direction to dog than to car in the toy table.
Cat points closer in direction to dog than to car in the toy table.

The rays begin at the same origin. Both axes use 60 SVG units per numerical coordinate, so cat [2,1], dog [2,2] and car [0,3] occupy data-derived positions. The mint arc marks cat's angular gap to dog, about 18.435°, compared with about 63.435° to car. The axes are arbitrary learned coordinates with no assigned human meaning.

Cosine compares directions by dividing the dot product by the product of vector lengths. Cat–dog gives 6/(58)≈0.9486836/(\sqrt5\sqrt8)\approx0.948683; cat–car gives 3/(5×3)≈0.4472143/(\sqrt5\times3)\approx0.447214. Thus this table ranks dog closer in direction. This is a property of the declared fixture, not evidence that a real model has learned useful semantics.

Learn From Contexts

Cat and dog share contexts in the declared count table.
Cat and dog share contexts in the declared count table.

Rows name words; columns name context words. Entries count adjacent pairs in a synthetic corpus: pet–cat 3, feed–cat 2, pet–dog 2, feed–dog 3 and road–car 5; all other displayed counts are zero. The outline groups the shared nonzero contexts for cat and dog. Similar usage patterns provide evidence for learning related representations, though relatedness need not mean synonymy.

These counts do not derive our earlier illustrative E. GloVe, section 3, fits a weighted objective over corpus co-occurrence statistics, while word2vec, section 3, learns through local context prediction. Objectives and data change the resulting geometry; there is no fixed semantic coordinate table waiting to be recovered.

Measure Euclidean Gaps

Euclidean gaps in the toy table are 1 and root eight.
Euclidean gaps in the toy table are 1 and root eight.

Keep the exact positions and axis scale from the ray plot, but connect endpoints instead. The vertical cat–dog segment has length 1 in coordinate units. The diagonal cat–car segment has length (2−0)2+(1−3)2=8≈2.828427\sqrt{(2-0)^2+(1-3)^2}=\sqrt8\approx2.828427. At 60 SVG units per coordinate, those lengths are 60 and about 169.706.

Euclidean distance measures the full difference, including magnitude. Cosine isolates direction for nonzero vectors. Neither metric guarantees useful semantic judgments, and distances in these arbitrary embedding units should not be interpreted as a universal meaning scale.

Pay For Table Width

Tripling table width triples its storage at fixed vocabulary and precision.
Tripling table width triples its storage at fixed vocabulary and precision.

For 10,000 word types at four bytes per value, 100 dimensions need 4,000,000 bytes and 300 need 12,000,000 bytes: 4 MB and 12 MB using decimal units. Bars share 25 SVG units per MB. This counts the table alone, excluding optimizer state, activations and other model costs.

A larger table costs memory but does not guarantee better task quality. Choose width using held-out evaluation and resource constraints. The example provides no universal recommended dimension.

Retrieve A Neighbor

Cosine retrieval ranks dog ahead of car for the toy cat query.
Cosine retrieval ranks dog ahead of car for the toy cat query.

Use cat's vector as a query and exclude cat itself from the candidates. Sort descending cosine: dog .948683 ranks first, car .447214 second. The bar lengths use 200 SVG units per cosine unit, approximately 189.737 and 89.443. The digits identify rank, while lengths show score; a score is not a probability.

This illustrates a nearest-neighbor use of the same table, not a claim that its synthetic coordinates were learned. Real retrieval quality depends on training evidence, normalization choices and the intended notion of relevance.

Separate Scale From Direction

Positive rescaling preserves a vector's cosine direction.
Positive rescaling preserves a vector's cosine direction.

This plot uses a declared scale of 50 SVG units per coordinate to fit [4,2]. Cat [2,1] and its doubled vector [4,2] lie on exactly the same ray. The circle marks the original endpoint and the diamond the doubled endpoint. The original length is √5 and the doubled length is 2√5.

For a positive scalar a and nonzero vectors, cos⁡(av,q)=cos⁡(v,q)\cos(av,q)=\cos(v,q): both numerator and denominator gain the same factor. Euclidean distances generally change. Negative scaling reverses direction, and cosine for a zero vector is mathematically undefined without an implementation-specific convention.

Pooling Loses Order

Averaging word vectors loses their order.
Averaging word vectors loses their order.

Both ordered inputs converge to the same mean vector: ([2,1]+[2,2])/2=[2,1.5]([2,1]+[2,2])/2=[2,1.5]. Reversing their order cannot change addition. Averaging is a simple way to pool word vectors, but it cannot distinguish sequences containing the same vectors in different orders. These two-word lists are a controlled algebra example, not full sentence embeddings or a claim that every sentence encoder averages its inputs.

Where It Breaks

A static bank lookup returns the same row in different contexts.
A static bank lookup returns the same row in different contexts.

The bank occurrences beside cash and river have different intended senses, yet both select the identical row v. The converging lookup paths visibly discard which surrounding word appeared. Adding more dimensions alone cannot make this static lookup occurrence-dependent; its input is still the same ID.

A static vector may contain mixed usage information and need not sit at a geometric midpoint between senses. Downstream context processing can disambiguate it. A contextual encoder computes a representation from surrounding tokens, as in BERT, although context access alone guarantees no accuracy. Use a context-sensitive representation when the current sense matters rather than expecting a fixed word row to switch meanings.

The Quick Version

  • One-hot identities have equal pairwise distances.
  • An embedding ID selects a learned table row.
  • Toy directions yield cat–dog cosine .949 and cat–car .447.
  • Shared contexts supply evidence, not predetermined coordinates.
  • Euclidean distance measures magnitude as well as direction.
  • Table storage grows linearly with vocabulary and width.
  • Cosine retrieval ranks candidates without producing probabilities.
  • Positive rescaling preserves cosine direction for nonzero vectors.
  • Averaging word vectors discards their order.
  • A static word row cannot adapt to the current sense.