Skip to content
AI360XpertBeta
Core ML

Visual explainer

Embedding Space

Trace coordinates, cosine, distances and normalization in one toy embedding table, then see why a static vector cannot adapt to context.

Spelling distance ranks car ahead of kitten for cat.
Spelling distance ranks car ahead of kitten for cat.

Counting letter edits puts car closer to cat than kitten. An embedding gives words numerical coordinates so you can compare the relationships captured by a trained model. Its distances are useful only when the training task and chosen metric match your question.

The bars count single-character insertions, deletions and substitutions: cat → car takes one; cat → kitten takes five. One possible five-edit route substitutes c→k and a→i, then inserts t, e and n. This is a spelling measure, not a semantic judgment.

Coordinates Place Words

A stored coordinate pair places each word at a point.
A stored coordinate pair places each word at a point.

Our deliberately synthetic table assigns cat = (2, 1), kitten = (1.8, 1.2), and car = (0.2, 2.8). These aren't outputs from a real model. Both plot axes use the same scale; the dashed guides locate cat's two coordinates. Learned dimensions needn't correspond to named human concepts.

Training supplies the table. For example, GloVe fits word vectors using corpus co-occurrences. Our drawing begins after training and explores the resulting geometry. Real embeddings often have many more than two coordinates.

Compare Directions

Vectors a and b form a 45-degree angle.
Vectors a and b form a 45-degree angle.

This separate mathematical pair makes the angle easy to check: a = (2, 1), b = (1, 3). Rays start at the same origin; the arc spans the angle between them. Cosine similarity divides their dot product by their lengths:

cos⁡θ=a⋅b∥a∥∥b∥=2×1+1×3510=12≈0.707107.\cos\theta = \frac{a\cdot b}{\|a\|\|b\|} = \frac{2\times1+1\times3}{\sqrt5\sqrt{10}} = \frac1{\sqrt2}\approx0.707107.

Thus θ = 45°. This calculation measures direction, with no claim that these two vectors represent particular words. The ordinary geometric definition requires nonzero vectors.

Reuse an Offset

An idealized shared offset completes the king–queen analogy.
An idealized shared offset completes the king–queen analogy.

In another explicitly idealized table, man = (0, 0), woman = (1, 1), king = (2, 0), queen = (3, 1). The two diagonal displacements are identical: woman − man = (1, 1), and king + (1, 1) = queen. Dashed horizontal edges show the matching relation across the parallelogram.

The GloVe paper discusses analogies of this form. Real learned offsets are approximate, depend on the corpus and can encode social biases. This toy equality is neither a universal rule nor evidence of a single axis that defines gender. The zero point here is used for offset arithmetic, not cosine.

Measure Endpoint Gaps

Kitten is the nearest candidate under this table's Euclidean metric.
Kitten is the nearest candidate under this table's Euclidean metric.

Return to the cat/kitten/car table. Euclidean distance is the straight-line gap between endpoints. Both bars use one common ruler in coordinate units:

d(cat,kitten)=(2−1.8)2+(1−1.2)2=0.08≈0.283,d(\text{cat},\text{kitten})=\sqrt{(2-1.8)^2+(1-1.2)^2}=\sqrt{0.08}\approx0.283, d(cat,car)=(2−0.2)2+(1−2.8)2=6.48≈2.546.d(\text{cat},\text{car})=\sqrt{(2-0.2)^2+(1-2.8)^2}=\sqrt{6.48}\approx2.546.

This table ranks kitten first. The result follows from the declared coordinates; geometry alone cannot establish that a learned table captures every meaning you care about.

Length Can Change the Answer

Scaling a vector changes its Euclidean gap but leaves its direction.
Scaling a vector changes its Euclidean gap but leaves its direction.

Here the circle marks a = (1, 1), and the diamond marks b = (3, 3). The common ray shows identical direction, so cosine is 1. The parallel bracket measures the nonzero endpoint gap, √(2² + 2²) = √8 ≈ 2.828.

Cosine ignores positive rescaling; Euclidean distance retains it. Choose the metric used by the model's training or evaluation objective. A vector's length doesn't automatically mean word frequency, confidence or document size.

Normalize the Length

Unit normalization moves vectors onto one common circle.
Unit normalization moves vectors onto one common circle.

Divide each nonzero vector by its length. The rings show normalized cat ≈ (0.894427, 0.447214) and car ≈ (0.071247, 0.997459). Each lies on the radius-one circle. Dashed rays indicate their directions and continue past the normalized points; their outer ends do not encode original vector lengths.

Normalization removes magnitude information and leaves direction. It adds preprocessing and stored-vector constraints when building a retrieval system; apply the same convention to queries and candidates.

Rankings Now Agree

Cosine and Euclidean rankings agree for unit vectors.
Cosine and Euclidean rankings agree for unit vectors.

The ring is cat, the filled point is kitten, and the diamond is car, now all on the same unit circle. The short mint chord connects cat to kitten. Their normalized coordinates give cosines about 0.992278 and 0.509802 respectively.

For unit vectors u and v, squared Euclidean distance is 2 − 2(u·v). Consequently, maximizing cosine and minimizing Euclidean distance produce the same ranking. The chord lengths here are about 0.124275 for kitten and 0.990149 for car. This agreement needs unit normalization; it doesn't hold for arbitrary raw lengths.

Rotate the Coordinates

A shared rotation preserves distances and cosine.
A shared rotation preserves distances and cosine.

Rotate the original table by 90° with (x₁, x₂) → (−x₂, x₁). Cat becomes (−1, 2), kitten (−1.2, 1.8), and car (−2.8, 0.2). Both panels keep the same units: circle = cat, dot = kitten, diamond = car. The right panel includes negative horizontal coordinates.

A common orthogonal rotation preserves dot products and lengths, so all previous distances and cosine rankings survive. For a fresh check, cat–car remains √6.48. Individual coordinates changed completely; the relationships didn't. Arbitrary rescaling of just one axis would not preserve this geometry.

Where It Breaks

Two bank contexts still select the same static vector.
Two bank contexts still select the same static vector.

A static word-type table selects the same bank row for “river bank” and “loan bank.” The illustrative row (1, 2) cannot change its neighbors according to the current sentence. Nothing requires it to sit exactly halfway between the two senses.

Context-aware processing can produce different occurrence representations; see word embeddings and Transformers. Even contextual geometry needs evaluation on the intended task. Proximity in a plot is not a guarantee of semantic correctness.

The Quick Version

  • Letter edits compare spelling; embeddings encode learned relationships.
  • Coordinates determine distances and directions.
  • Cosine normalizes the dot product of nonzero vectors.
  • Analogy offsets are approximate properties of particular trained tables.
  • Euclidean gaps retain magnitude differences.
  • Unit normalization makes cosine and Euclidean rankings agree.
  • Shared rotations preserve the geometry.
  • Static word vectors cannot adapt to each occurrence's context.