Gen AI

Embeddings

A model maps discrete things like words or documents into a continuous vector space where distance carries meaning, so semantically related items land close together.

A model maps discrete things into a continuous space where distance carries meaning, so a query about wireless headphones lands near matching products and far from unrelated ones

Why Does This Exist?

Ask a database to find documents that mean the same thing as a user's question, and keyword matching immediately falls short: "how do I reset my password" and "forgot login credentials" share almost no words, yet they're asking the same thing. A system that can only match exact terms treats those two phrases as unrelated, because as strings, they mostly are.

Embeddings solve this by moving the comparison out of the space of words and into a space of meaning. A trained model maps any piece of text — a word, a sentence, a whole document — into a fixed-length vector of numbers, positioned so that texts with similar meaning land close together in that vector space, regardless of which specific words they used to express it. Once "reset my password" and "forgot login credentials" are both just points in space, the question "are these related" becomes the question "how far apart are these two points" — a computation that takes microseconds and doesn't care that the two phrases share almost no vocabulary.

Think of It Like This

A map where nearby cities are actually related, not just alphabetically close

Imagine every product in a massive catalog got assigned a location on a giant map, not based on the product's name alphabetically, but based on what it actually is and does. Wireless earbuds and bluetooth headphones would end up as neighboring dots on this map, close together, even though their names share barely a letter. A garden hose would sit somewhere entirely different, because nothing about what it is relates to audio equipment, regardless of alphabetical proximity.

Embeddings build exactly this kind of map, except the "location" is a vector of numbers rather than latitude and longitude, and the model that assigns positions has learned, from enormous amounts of text, what actually relates to what. Distance on this map is meaningful by construction — that's the entire property the map was built to have.

How It Actually Works

From text to a point in space

An embedding model — itself a trained neural network, often built on the same transformer architecture underlying language models generally — takes a piece of text as input and outputs a fixed-length vector, typically somewhere between 256 and a few thousand dimensions. Every input, no matter how long or short, produces a vector of the same length: a single word and a full paragraph from the same model both come out as, say, 1,024 numbers. What varies is where in that 1,024-dimensional space each one lands.

Why distance means something here

The property that makes this useful — and the property that the model is specifically trained to produce — is that the geometry of the space carries semantic information. Two pieces of text with similar meaning are positioned close together; two with unrelated meaning are positioned far apart. Nothing about a vector's individual numbers is directly interpretable (dimension 402 isn't labeled "formality" or anything else a person would recognize), but the relationships between vectors are exactly what the model was optimized to get right. Measuring how close two embeddings are — using a similarity metric like cosine similarity — is how that semantic relationship gets turned into an actual, usable number.

One vector per unit of meaning, chosen deliberately

Embeddings can represent text at different granularities depending on what a system needs: a single word, a sentence, an entire document. For retrieval systems specifically, the common choice is one vector per chunk — a passage-sized piece of a larger document, small enough to be specific, large enough to carry real context, a decision covered directly on chunking strategies. Whatever granularity is chosen, storing enough of these vectors to search over efficiently is the job vector databases exist to do.

The one rule that governs everything downstream

Every embedding used in a comparison must come from the same model, at the same version. Two vectors from different models occupy entirely different, unrelated coordinate systems — a distance computed between them is not meaningful, even though the arithmetic runs without any error. This single constraint governs nearly every operational decision covered later in this band: upgrading an embedding model means re-embedding an entire corpus, not just the new documents arriving after the upgrade.

Show Me the Code

A tiny illustrative embedding function and a distance check, showing exactly what "close in the space" means numerically.

1import numpy as np2
3
4def toy_embed(text: str, dim: int = 8) -> np.ndarray:5    """Not a real model -- a deterministic stand-in so the same input always maps to the same vector."""6    rng = np.random.default_rng(abs(hash(text)) % (2**32))7    return rng.normal(size=dim)8
9
10def cosine(u: np.ndarray, v: np.ndarray) -> float:11    return float(u @ v / (np.linalg.norm(u) * np.linalg.norm(v)))12
13
14headphones = toy_embed("wireless bluetooth headphones")15earbuds = toy_embed("wireless bluetooth headphones")     # identical text -> identical vector16hose = toy_embed("50-foot garden hose")17
18print(round(cosine(headphones, earbuds), 4))   # -> 1.0 -- same text, same point in space19print(round(cosine(headphones, hose), 4))       # -> near 0 -- a real model would separate these further

A real embedding model would place headphones and a genuinely different phrase like "noise-cancelling earbuds" close together despite sharing few words — this toy version only demonstrates that identical input always lands at the identical point, which is the one property any embedding function must guarantee.

Watch Out For

Mixing embeddings from two different models in one index

Storing vectors from an old embedding model alongside vectors from an upgraded one, without re-embedding the old documents, produces an index that returns confidently wrong results. Nothing errors — the dimensions match, the distances compute, the query returns a ranked list — but distances between vectors from different models measure nothing meaningful, since the two models never agreed on what any given direction in space represents.

Reading individual dimensions as meaningful attributes

Because embeddings are positioned by meaning overall, it's tempting to assume some specific dimension corresponds to some specific human-readable attribute — sentiment, formality, topic. Almost never true for a general-purpose embedding model: the space's structure emerges from training, and there's no guarantee any individual axis aligns with a concept a person would recognize. What's meaningful is relative position and distance, not any single coordinate read in isolation.

The Quick Version

Embeddings map text into a continuous vector space where semantic similarity corresponds to geometric closeness.
Every input produces a fixed-length vector regardless of the input's own length, and only relative position carries meaning.
Distance between two embeddings is measured with a similarity metric, most often cosine similarity for text embeddings.
Embeddings from different models occupy unrelated spaces — comparing across models produces meaningless distances without erroring.
Retrieval systems commonly embed at the chunk level, one vector per passage-sized piece of a document.