Vector Embeddings

One-hot puts every category the same distance apart. An embedding is a short list of numbers where distance means similarity, so search becomes ranking.

The same three vectors rank differently before and after normalising: raw dot product picks the longer rival at 40 over 24, while cosine picks the true neighbour at 0.96 over 0.72

Why Does This Exist?

One-hot encode 50,000 products and then ask which two are alike. You get the same answer for every pair. Any two one-hot columns are orthogonal, so any two categories sit exactly $\sqrt{2}$ apart. A phone charger and a phone case land as far apart as a charger and a tent. The similarity was never written down.

An embedding replaces that with a short dense vector whose geometry carries the meaning. Close in space means close in meaning, and you can rank a million candidates by one number.

This page treats embeddings as data: what you store, normalise, index, version and watch in production. How an encoder learns them is a training story. The count-to-encoding decision that comes before this is categorical encoding, the sparse text baseline is text feature representation, and the geometry sequel is latent spaces and manifold learning.

Basics first: cardinality is the number of distinct values in a column. Two values fit one bit, a dozen fit one-hot, 50,000 do not. If that trade is new, read categorical encoding first. Everything below assumes you know why a wide one-hot block breaks.

Think of It Like This

Spice jars arranged by smell

Imagine a market stall with 200 spice jars. One arrangement sorts them by the order they arrived. Cumin sits next to vanilla because those boxes came on the same truck. The position tells you nothing about taste. That is one-hot.

Now the seller rearranges by smell. Warm spices cluster at one end, bright citrus peels at the other, smoky chilies together in a corner. Neighbours taste alike, far jars do not, and nobody wrote a rule that says so. The position is the information. That is an embedding.

Where it stops: move the stall to a new supplier and every jar gets a new spot under a new nose. Old notes like third shelf from the left now point at the wrong jar, though they still look valid. That is what swapping your embedding model does to a stored index.

How It Actually Works

What you store

A fixed-length array of floats, usually 32 to 3072 wide. A category table inside your own model might use 32. A small sentence encoder gives 384. A hosted model often gives 1536. The width is set once and then fixed, because every row, every index and every query assumes it.

What makes it an embedding is not the file format. It is that distance was fitted to mean something. A vector store will hold random numbers without complaint, so the meaning has to come from the encoder that produced them.

You will meet four kinds. Learned category vectors trained as a table inside your model. Static word vectors with one vector per token, so bank keeps one address for both meanings. Contextual vectors with one address per occurrence, so the two banks differ. And one vector per sentence or document, which is what retrieval ranks and what this page assumes.

Normalise first, or length votes

Divide a vector by its L2 norm and you get a unit vector. Do it on both sides and the dot product becomes the cosine:

\hat{a} \cdot \hat{b} = \frac{a \cdot b}{\lVert a \rVert \, \lVert b \rVert} = \cos\theta

Here $a$ and $b$ are two embeddings, $\lVert a \rVert$ is the length of $a$ , $\hat{a} = a / \lVert a \rVert$ is the same direction scaled to length 1, and $\theta$ is the angle between them.

Most indexes assume you did this. Score unnormalised vectors by inner product and a long document outranks a short relevant one on length alone. Normalise at write time and at query time, and only direction counts. The full menu of distance choices lives in distance and similarity metrics.

Width, bytes and the index

Memory is one line: rows times dimensions times 4 bytes for float32. Ten million vectors at 1536 wide need $10^7 \times 1536 \times 4 = 61{,}440{,}000{,}000$ bytes, about 61GB before the index adds overhead. That is why int8 quantisation exists: one byte per dimension turns 61GB into about 15GB, usually for a point or two of recall. Binary packs one bit per dimension, about 1.9GB here, with a larger accuracy cost, so teams rerank the top few hundred in full precision.

Search is the second bill. Exact search scans every vector per query, so it grows with the corpus (computational complexity). HNSW walks a navigable graph instead. IVF clusters the space and visits a few cells. Both trade recall for latency. Measure recall at k against exact search on a few thousand queries, and keep it near 0.95 at $k = 10$ before you trust the speed.

One model, one space

Two vectors compare only when they came from the same model at the same version. Change the model and every stored row needs a fresh pass, because the new axes have no relation to the old ones. Shapes still match, so nothing errors.

Three limits to say plainly. No dimension has a name. Reading axis 402 as formality is reading noise. Cosine is not relevance: two documents on one topic score high even when neither answers the question, which is why serious retrieval adds a reranker. And the encoder bias is now geometry, so stereotypes return as near neighbours.

Inputs move while the encoder stays frozen (data drift and distribution shift). New topics land in thin regions where neighbours are far and close to arbitrary, and the symptom is a slow slide in clicks with no code change.

Worked example

Take query $q = (3, 4, 0)$ , a near neighbour $a = (4, 3, 0)$ and a longer rival $b = (0, 10, 5)$ .

Lengths: $\lVert q \rVert = \sqrt{9 + 16} = 5$ , $\lVert a \rVert = \sqrt{16 + 9} = 5$ , $\lVert b \rVert = \sqrt{0 + 100 + 25} = \sqrt{125} = 11.180$ .

Cosines: $q \cdot a = 12 + 12 = 24$ , so $24 / (5 \times 5) = 0.96$ . Then $q \cdot b = 0 + 40 + 0 = 40$ , so $40 / (5 \times 11.180) = 0.716$ . Cosine ranks $a$ first, which matches direction.

Raw dot products skip the division: 24 versus 40, so $b$ wins. Nothing about meaning changed. Length voted, and length is all the raw dot product can see.

Code

1import numpy as np2
3query: np.ndarray = np.array([3.0, 4.0, 0.0])4near: np.ndarray = np.array([4.0, 3.0, 0.0])5long_doc: np.ndarray = np.array([0.0, 10.0, 5.0])6
7def unit(v: np.ndarray) -> np.ndarray:8    return v / np.linalg.norm(v)9
10print(round(float(unit(query) @ unit(near)), 3))  # -> 0.9611print(round(float(unit(query) @ unit(long_doc)), 3))  # -> 0.71612print(round(float(query @ near), 1))  # -> 24.013print(round(float(query @ long_doc), 1))  # -> 40.014print(round(10_000_000 * 1536 * 4 / 1e9, 2))  # -> 61.44

One missing call to unit flips the winner from the true neighbour to the longer rival.

Watch Out For

Two model versions inside one index

You upgrade the encoder and re-embed new rows as they arrive, leaving old rows in place to avoid a six hour backfill.

Nothing breaks. Dimensions match, writes succeed, queries return ten neighbours with scores between minus 1 and 1. Those cross-version scores measure nothing, because the two spaces put their axes in unrelated places. About half your results turn to noise and the only symptom is relevance slipping a little each week.

Put the model version in the index name itself, not in a metadata field someone forgets to filter. An upgrade then means what it is: build a new index, backfill every row, swap, drop the old one.

Inner-product search on vectors nobody normalised

The encoder returns raw vectors, the store defaults to inner product, and both get left as they are.

Now length ranks. A 4,000 word page carries a larger vector than a 200 word answer, so it outscores the answer on nearly every query. Your top results skew long and generic while the scores still look like similarities. The worked example above is the whole bug in three integers: cosine says 0.96 against 0.716, raw dot says 24 against 40, and they crown different winners.

Normalise at write time and at query time, then inner product and cosine agree. Check one stored norm first. If it is not 1.0, you found it.

The Quick Version

One-hot puts every pair at the same distance, so it stores no similarity. An embedding stores similarity as geometry.
This page is about embeddings as data to store and serve. Training the encoder is a separate story.
Normalise and the dot product becomes a cosine. Skip it and length enters your ranking.
Memory is rows times dims times 4 bytes. Ten million at 1536 wide is about 61GB, about 15GB in int8, about 1.9GB binary with a rerank.
Exact search scans everything. HNSW and IVF trade recall for latency, so hold recall near 0.95 at 10.
Vectors compare only inside one model version. An upgrade means re-embedding everything.
No axis has a name, cosine is not relevance, and encoder bias ships as geometry.
The encoder is frozen so stored rows do not move. Your inputs drift into thin regions instead.