Word Embeddings

Word embeddings map discrete words to continuous vectors of real numbers, allowing machine learning models to capture and mathematically manipulate semantic meaning.

Word embeddings position semantically related words close together in a dense vector space, allowing relationships like 'king - man + woman = queen' to emerge as simple geometric translations.

Why Does This Exist?

For decades, natural language processing relied on discrete representations of text. A word was treated as an isolated, categorical symbol. The most common approach was "one-hot encoding," where a vocabulary of $V$ words is represented as vectors of length $V$ . If your vocabulary has 10,000 words, the word "king" might be a vector with a $1$ at index 4,032 and $0$ s everywhere else. The word "queen" might be a $1$ at index 7,121.

This approach is highly inefficient and logically flawed for understanding language. In a one-hot encoded space, every word is mathematically equidistant from every other word. The dot product between the vectors for "king" and "queen" is exactly the same as the dot product between "king" and "screwdriver" — zero. The model is given absolutely no mathematical hint that "king" and "queen" are semantically related, nor that "running" and "ran" are grammatically linked. Every time the model encounters a new word, it has to learn its properties from scratch, completely independently of words it has already seen.

Furthermore, as the vocabulary grows, the vectors become massively sparse, consuming enormous amounts of memory while providing almost zero useful structural information.

Word embeddings were developed to solve this fundamental representation problem. Instead of sparse, high-dimensional, discrete vectors, word embeddings map words to dense, relatively low-dimensional vectors (often between 100 and 1000 dimensions) of continuous real numbers. In this continuous vector space, the geometric distance and direction between words correspond directly to their semantic and syntactic relationships. Models can finally do math on meaning.

Think of It Like This

Plotting locations on a map

Imagine trying to describe the locations of cities using only an alphabetical list. "Austin" is item 1, "Boston" is item 2, and "Chicago" is item 3. If you ask someone "How far is Austin from Boston?", the alphabetical list is useless. They have no relationship other than their starting letter. This is what one-hot encoding does to words.

Now, imagine dropping those cities onto a two-dimensional map with latitude and longitude coordinates. Austin is at (30.2, -97.7), and Boston is at (42.3, -71.0). Suddenly, you can calculate distance. You can see that Boston is close to New York, and that moving strictly south from Boston takes you to Miami.

Word embeddings do exactly this, but instead of physical cities mapped onto two dimensions (latitude and longitude), they are semantic words mapped onto hundreds of dimensions. Each dimension represents a learned, abstract feature of meaning (like "royalty," "gender," "plurality," or "color"). By placing words in this coordinate space, you give the computer a "map of meaning" where distance represents similarity and direction represents semantic relationships.

How It Actually Works

Word embeddings are not manually designed; they are learned from vast amounts of text through a self-supervised process. The foundational insight that makes this possible is known as the distributional hypothesis, summarized by the linguist John Rupert Firth in 1957: "You shall know a word by the company it keeps." If two words frequently appear in the exact same surrounding context, they likely share a similar meaning.

Here is how algorithms like Word2Vec learn these embeddings:

1. Defining the context window

The algorithm reads through a massive corpus of text, sliding a "context window" over the words. For every target word it focuses on, the context window captures the surrounding words (e.g., two words before and two words after). For the sentence "The quick brown fox jumps over the lazy dog", if the target word is "fox", the context words might be "quick", "brown", "jumps", and "over".

2. Setting up the prediction task

The algorithm sets up a fake supervised learning task. In the Continuous Bag of Words (CBOW) architecture, the model tries to predict the target word given its surrounding context words. In the Skip-gram architecture, the model takes the target word and tries to predict the surrounding context words. The actual accuracy of this prediction task isn't the final goal; the task exists purely to force the model to learn useful internal representations.

3. Updating the vector weights

Initially, every word in the vocabulary is assigned a random dense vector. As the model attempts the prediction task, it makes errors. It uses backpropagation to adjust the weights of its internal vectors slightly, moving the vector for the target word closer to the vectors of words that tend to appear in similar contexts, and pushing it away from words that do not.

4. Convergence and semantic geometry

After passing over millions or billions of words, the vectors stabilize. The result is a vector space where words with similar contexts end up clustered together. Because of the way the vector arithmetic works out, the space captures not just similarity, but relational directions. The vector transition from "man" to "woman" becomes roughly parallel to the transition from "king" to "queen" or "uncle" to "aunt". Thus, the famous vector algebra equation emerges:

\vec{v}_{\text{king}} - \vec{v}_{\text{man}} + \vec{v}_{\text{woman}} \approx \vec{v}_{\text{queen}}

This dense representation is then extracted and used as the fundamental input layer for almost all modern downstream NLP tasks. Rather than feeding raw strings or one-hot vectors into a neural network, text is tokenized, mapped to these pre-trained dense vectors, and the network immediately starts with a rich, mathematical understanding of the words it is processing.

Code

This is a minimal demonstration of how word embeddings capture semantic relationships using a small pre-trained vector set. We use cosine similarity to measure the angle between vectors; a value closer to 1 means the vectors point in the same direction.

1import numpy as np2
3# Toy embeddings (normally 300+ dimensions, truncated here for readability)4# Dimensions might vaguely represent: [royalty, gender(positive=female), human]5vocab = {6    "king":  np.array([ 0.95, -0.80,  0.90]),7    "queen": np.array([ 0.97,  0.85,  0.92]),8    "man":   np.array([ 0.05, -0.82,  0.95]),9    "woman": np.array([ 0.02,  0.86,  0.91]),10    "apple": np.array([-0.10,  0.05, -0.90])11}12
13def cosine_similarity(v1: np.ndarray, v2: np.ndarray) -> float:14    dot_product = np.dot(v1, v2)15    norm_v1 = np.linalg.norm(v1)16    norm_v2 = np.linalg.norm(v2)17    return float(dot_product / (norm_v1 * norm_v2))18
19# 1. Similarity check20print(f"King & Queen: {cosine_similarity(vocab['king'], vocab['queen']):.3f}")21# -> King & Queen: 0.354 (Related by royalty/human, opposite in gender)22print(f"King & Apple: {cosine_similarity(vocab['king'], vocab['apple']):.3f}")23# -> King & Apple: -0.638 (Highly unrelated)24
25# 2. Vector Arithmetic: King - Man + Woman = ?26target_vector = vocab['king'] - vocab['man'] + vocab['woman']27
28# Find closest word to the resulting target vector29best_word = None30best_sim = -1.031
32for word, vector in vocab.items():33    if word in ["king", "man", "woman"]:34        continue35    sim = cosine_similarity(target_vector, vector)36    if sim > best_sim:37        best_sim = sim38        best_word = word39
40print(f"King - Man + Woman approximates: {best_word} (sim: {best_sim:.3f})")41# -> King - Man + Woman approximates: queen (sim: 0.999)

Watch Out For

Conflating static embeddings with contextual embeddings

Classic models like Word2Vec and GloVe produce static embeddings: a word like "bank" has exactly one vector representation, regardless of whether it appears in "river bank" or "bank account". This limits their ability to handle polysemy (words with multiple meanings). Modern transformer architectures use contextual embeddings, where the vector for "bank" is computed on the fly based on the surrounding sentence. Static embeddings are still useful for lightweight tasks or initial layers, but they do not dynamically adapt to context.

Inheriting bias from the training corpus

Because word embeddings learn from historical human text, they faithfully encode historical human biases. If the training corpus frequently associates "programmer" with "he" and "nurse" with "she", the resulting vector space will reflect this. Running the arithmetic $\vec{v}_{\text{doctor}} - \vec{v}_{\text{man}} + \vec{v}_{\text{woman}}$ on uncorrected embeddings famously returns $\vec{v}_{\text{nurse}}$ . Using these vectors blindly in downstream systems, like resume screening tools, can mathematically institutionalize prejudice.

The Quick Version

Word embeddings represent words as dense, continuous vectors of real numbers rather than sparse, discrete symbols.
They are trained on massive text corpora using the distributional hypothesis: words appearing in similar contexts share similar meanings.
The resulting vector space maps semantic similarity to geometric distance, allowing models to use dot products and cosine similarity to evaluate relationships.
Vector arithmetic within this space reveals complex analogies, such as "king" relates to "man" as "queen" relates to "woman".
While powerful, static embeddings compress all meanings of a word into a single vector and directly inherit societal biases present in their training data.