Word Embeddings

Word embeddings give each word a list of numbers that places meaning on a map. Close words sit near each other, so similarity becomes distance you can measure.

Cat sits near dog with cosine 0.949 while car sits far at 0.447, so meaning becomes angle and distance you can compute.

Why Does This Exist?

One hot vectors give every word its own axis and learn nothing. With a vocabulary of size $V$ , here $V$ is the count of words you track, king is a 1 in one slot and queen is a 1 in another. Their dot product is zero, exactly like king and screwdriver. The model gets no hint that royalty belongs together, so each word trains alone and rare words starve.

Word embeddings replace those sparse slots with short dense lists. Each word owns $d$ real numbers, where $d$ is the width you pick, often 100 to 300. Words that keep similar company land near each other, so distance starts to mean something and the network begins with a map instead of a blank.

This page covers that map. The cut into units lives in tokenization, the sentence level version in sentence embeddings, and the ruler itself in similarity metrics.

The one idea you need first

Cosine similarity measures angle, not length. It divides the dot product by both lengths, so 1 means same direction, 0 means right angle, and minus 1 means opposite. Close in angle means close in meaning here, which is the full prerequisite.

Think of It Like This

Plotting locations on a map

An alphabetical list tells you nothing about travel. Austin follows Atlantic City by letter, yet sits far by road. You cannot plan a trip from sort order.

Drop the same cities on a map with two numbers each. Now Boston sits near New York, Miami sits south, and distance answers the question the list could not. Direction answers more: south from Boston points toward Miami.

Embeddings are that map for words, with hundreds of learned axes instead of two. Royalty, gender and tense become directions you can walk. Where it stops: a map has true north while learned axes do not. No single axis means royalty, so you read distances and shifts, never coordinates alone.

How It Actually Works

One hot keeps every pair equally far

With $V = 10000$ , cat, dog and car each own a 10000 slot vector with one 1. Cat dot dog is 0. Cat dot car is 0. Both gaps equal the square root of 2, about 1.414. The table knows spelling slots, not meaning, and each new word starts from zero.

Dense vectors learn from company

You shall know a word by the company it keeps. Slide a small window across text, then nudge words that share neighbors toward each other. Prediction is the excuse, placement is the product. After millions of windows, pet words cluster, road words drift elsewhere, and the shift from man to woman runs parallel to the shift from king to queen.

Worked closeness

Use two numbers per word so you can check the angle by hand. Let $a$ dot $b$ mean multiply and add, and let $|a|$ mean the length.

Cat is $[2, 1]$ , dog is $[2, 2]$ , car is $[0, 3]$ . Dot of cat and dog is $2 \times 2 + 1 \times 2 = 6$ . Lengths are $|cat| = \sqrt{5} \approx 2.236$ and $|dog| = \sqrt{8} \approx 2.828$ . Cosine is $6 / (2.236 \times 2.828) \approx 0.949$ , about 18 degrees. Near.

Dot of cat and car is $2 \times 0 + 1 \times 3 = 3$ . Length of car is $3$ . Cosine is $3 / (2.236 \times 3) \approx 0.447$ , about 63 degrees. Far. Those are the two scores in the diagram above, and the gap between 0.949 and 0.447 is the whole lesson.

The trade is plain. Static tables are cheap and fast, yet one vector must hold every sense of bank at once. Contextual vectors split senses per sentence, at the price of compute you pay on every read.

Code

Cosine for the worked pair, so 0.949 and 0.447 can be checked.

1import numpy as np2
3
4def cosine(a: np.ndarray, b: np.ndarray) -> float:5    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))6
7
8cat: np.ndarray = np.array([2.0, 1.0])9dog: np.ndarray = np.array([2.0, 2.0])10car: np.ndarray = np.array([0.0, 3.0])11print(f"{cosine(cat, dog):.3f}")  # -> 0.94912print(f"{cosine(cat, car):.3f}")  # -> 0.447

Same dot, same lengths, same two scores as the hand check. Near beats far by angle, not by spelling.

Watch Out For

One vector forced to hold two meanings

Symptom: bank sits halfway between river and money, near neither, and both queries return mush. A static table gives each spelling exactly one address, so polysemy has nowhere to go.

Fix: use contextual vectors when senses matter, or keep separate sense rows for the few ambiguous heads. Test bank style words as their own slice before you trust the map.

Bias baked into the geometry

Symptom: doctor minus man plus woman lands near nurse, not by logic but because the corpus paired those words that way. Ship the table into hiring or lending and you automate the past.

Fix: audit neighbor lists for role words, debias or constrain where needed, and never present analogy arithmetic as proof. Geometry reflects text, not truth.

The Quick Version

Word embeddings map each word to a short dense vector instead of a sparse one hot slot.
Training drags words that share neighbors to nearby points, so distance means similarity.
Cosine similarity turns angle into a score where 1 means same direction.
Worked case: cat and dog reach cosine 0.949 at 18 degrees while cat and car reach 0.447 at 63 degrees.
Shifts carry relations, as in king minus man plus woman landing near queen.
Static tables conflate senses and inherit corpus bias, so audit bank style words and role analogies.