Embedding Models

An embedding model turns text into vectors placed so related meanings land close together. You reach for one when search must match meaning instead of matching words.

Contrastive training scores whether the matching passage beats the lookalike by a margin, and here the lookalike wins so the loss is 0.379

Why Does This Exist?

"Reset my password" and "forgot login credentials" share almost no words, yet they ask the same thing. Keyword search sees two different strings and shrugs. An embedding fixes the representation: both texts become points in one vector space, and distance means relatedness. That is the promise embeddings makes.

But nothing about a standard language model guarantees that geometry. A model trained to predict the next word learns a space shaped by prediction, not by retrieval, so two paraphrases can sit far apart while a misleading lookalike sits near. That's why you train a second kind of model whose only job is placement: an encoder optimized so related texts land close and unrelated ones land apart, judged the way search will judge them.

This page covers that training and the one architectural choice that matters most, separate encoders for queries and documents. Scoring the resulting vectors is similarity metrics; putting them to work is semantic search.

The one idea you need first. A vector is a list of numbers, and cosine similarity scores the angle between two vectors from $-1$ (opposite) to $1$ (identical direction). Length is ignored; direction is the whole story.

Think of It Like This

Two doors into the same tasting room

A spice shop takes rush orders at a small street window ("dinner, fast") and bulk crates at the loading dock. Different doors, same tasting room inside, where jars sit side by side by flavor instead of by how they arrived. The staff trains one habit: when a shouted order and a crate taste alike, shelve them together; when a crate only sounds similar, push it to a far shelf.

Queries are the street window: short, casual, phrased as need. Documents are the loading dock: long, formal, written to inform. The embedding model is the staff habit, and the tasting room is the shared vector space. Separate doors because the inputs arrive in different shapes; one room because relevance is one judgment.

Where it stops: the staff can actually taste. An encoder never tastes meaning; it only positions by patterns in its training pairs, and the room's axes carry no human labels. Closeness is learned geometry, not understanding, which is why the wrong training pairs teach the wrong shelf order.

How It Actually Works

Close is trained, not found

Training is contrastive: each example names a query vector $q$ , its true match $p^{+}$ (p plus, the passage that answers it), and a rival $n$ (a passage that does not). With $\mathrm{sim}$ as cosine similarity and $m$ as the margin, the gap you demand between match and rival, the triplet loss is:

\mathcal{L} = \max(0,\ m + \mathrm{sim}(q, n) - \mathrm{sim}(q, p^{+}))

Read it as a bet. If the match already beats the rival by more than $m$ , the loss is zero and the example teaches nothing further. If not, the loss is positive and the gradient drags $q$ toward $p^{+}$ and away from $n$ . The margin exists because retrieval needs a cushion, not a hairline lead; a lead of 0.001 flips under noise.

The rivals decide what gets learned. Random pairings (cardiology versus cooking) teach almost nothing, since any space already separates them. A hard negative shares words with the query yet answers nothing, forcing the fine distinctions live search actually gets wrong. Mining those, usually from your own system's top results marked wrong by a person, is most of the work in practice.

The encoders reflect the asymmetry. A bi-encoder runs queries and documents through separately learned paths into one shared space, because a seven-word question and a seven-hundred-word passage are different jobs. That is the trade: two paths cost a little complexity and buy relevance where the inputs differ in shape. When the inputs match in shape, say paraphrase detection, a single symmetric encoder is the simpler and better pick.

Worked example: one triplet, scored by hand

Take $q = [1, 0]$ , its match $p^{+} = [0.8, 0.6]$ , and a lookalike $n = [0.95, 0.2]$ . Cosine similarity is the dot product divided by both lengths.

For the match: the dot product is $0.8$ , the lengths are $1$ and $\sqrt{0.64 + 0.36} = 1$ , so $\mathrm{sim}(q, p^{+}) = 0.800$ . For the lookalike: the dot product is $0.95$ , the lengths are $1$ and $\sqrt{0.9025 + 0.04} = \sqrt{0.9425} \approx 0.971$ , so $\mathrm{sim}(q, n) \approx 0.979$ .

The lookalike leads $0.979$ to $0.800$ , a gap of $-0.179$ where $+0.200$ is required. With margin $m = 0.2$ :

\mathcal{L} = \max(0,\ 0.2 + 0.979 - 0.800) = 0.379

Positive, so this triplet teaches: pull the match toward the query, push the lookalike away. The diagram above draws exactly this score, and the snippet below recomputes it.

Two honest limits. The pairs must mirror live traffic; when queries drift to new shorthand, the space goes stale. And the loss only knows the rivals you show it, so a diet of easy negatives trains a model that looks finished and ranks badly. When you can't collect real pairs, don't train yet: a rewritten query or a reranker usually wins there, and embedding fine-tuning is the sequel for the day your own pairs exist.

Code

The point of this snippet is the check, not the formula: recompute the worked example's two cosines and confirm the loss fires.

1import numpy as np2
3
4def cosine(a: np.ndarray, b: np.ndarray) -> float:5    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))6
7
8def triplet_loss(q: np.ndarray, pos: np.ndarray, neg: np.ndarray, margin: float = 0.2) -> float:9    return max(0.0, margin + cosine(q, neg) - cosine(q, pos))10
11
12q = np.array([1.0, 0.0])13pos = np.array([0.8, 0.6])14neg = np.array([0.95, 0.2])15
16print(round(cosine(q, pos), 3))  # -> 0.817print(round(cosine(q, neg), 3))  # -> 0.97918print(round(triplet_loss(q, pos, neg), 3))  # -> 0.37919print(cosine(q, pos) - cosine(q, neg) >= 0.2)  # -> False

The False on the last line is the whole story: the match trails by $0.179$ where the margin demands it lead by $0.2$ , so the loss stays positive and training keeps pushing.

Watch Out For

Grading a paraphrase model on retrieval duty

A model trained on same-shape pairs (two sentences, two reviews) can pass paraphrase tests and still rank badly on live search. The symptom is specific: demo queries phrased like documents rank fine, while short real user questions return confident nonsense. The cause is the asymmetry gap: the model never learned to bridge a seven-word need against a seven-hundred-word answer.

Check the model card before you commit: it should name query-document or asymmetric retrieval training, not just semantic similarity. If it doesn't, reach for a retrieval-trained bi-encoder instead of tuning prompts around the wrong geometry.

Swapping the model without re-embedding the index

A new embedding model scores better on every benchmark, you point the query path at it, and rank collapses. Nothing is broken except the assumption: every stored vector was placed by the old model, and two models' spaces are unrelated coordinate systems. A query from the new space compared against documents from the old one is noise.

Re-embed the entire corpus with the new model and version-pin it, queries and documents together, before the swap goes live. Budget that pass up front; on a large corpus it's the real cost of the upgrade, and approximate nearest neighbor search can only search whatever geometry you stored.

The Quick Version

An embedding model is an encoder trained so related texts land close together, judged the way search judges them rather than the way next-word prediction does.
Training is contrastive: each triplet names a query, its match, and a rival, and the loss fires unless the match leads by the margin.
On the worked triplet the lookalike leads 0.979 to 0.800, so with margin 0.2 the loss is 0.379 and training keeps pushing.
Hard negatives that share words but answer nothing do most of the teaching; random easy pairs teach nearly nothing.
Bi-encoders give queries and documents separate paths into one shared space, which wins whenever the two inputs differ in shape.
Never swap the model without re-embedding the whole index, since vectors from two models live in unrelated spaces.