Data Concepts

Vector Embeddings

One-hot says every category is exactly as unrelated to every other one. An embedding is a short list of numbers whose distances actually mean something.

Distance is the content: the query sits at cosine 0.96 from its nearest neighbour in the cluster and at −0.37 from the unrelated item, which is the one thing a one-hot column can never express

Why Does This Exist?

One-hot encode 50,000 merchants, then ask the representation a question. How similar are merchant 12 and merchant 4,001? You get the same answer for every pair you ask about. Any two one-hot columns are orthogonal, so any two categories sit exactly $\sqrt{2}$ apart. Two branches of one coffee chain are as far apart as a coffee shop and a tyre garage. The similarity isn't buried in there somewhere waiting to be found; it was never written down (categorical encoding).

An embedding swaps that for a short dense vector whose geometry carries the meaning. Similar things land close, and "close" is a number you can compute in a microsecond.

Scope, up front, because this word gets stretched. This page is about embeddings as data: the thing you store, version, index and monitor. How an encoder learns to produce good ones belongs to another band. Take the vectors as given and there's still a week of engineering in front of you.

Think of It Like This

Barcodes and shelves

Every book in a library has a barcode. Scan two of them and you learn nothing about whether the books are related, because the numbers were handed out in the order the books arrived. That's one-hot.

Now walk the shelves instead. Two books a hand's width apart are about roughly the same thing, and the ones four aisles over aren't. Nobody wrote that relationship down anywhere. It fell out of the shelving rule, and the position is the entire content.

Then the library reshelves under a new scheme. Every position you noted last year now points at an unrelated book, and nothing warns you, because "aisle 4, shelf 2" is still a perfectly valid address. That is exactly what swapping your embedding model does to a vector store.

How It Actually Works

What you're actually storing

A fixed-length array of floats. Usually somewhere between 128 and 3072 of them, and the width is a decision you make once and then live with. A small sentence encoder gives 384, a common hosted model gives 1536, a category lookup table inside your own model might give 32.

What makes it an embedding rather than an arbitrary array is that it was fitted so distance means something. Nothing in the file format records that, and a vector store will hold garbage without complaint.

The varieties you'll actually meet, sorted by what they cost you to operate:

A learned category embedding, trained as a lookup table inside your own model. 50,000 rows by 32 columns replaces a 50,000-wide one-hot block, and the table ships with the weights.
Static word vectors: one per token, context ignored, so bank gets a single vector covering both meanings.
Contextual token vectors: one per token per occurrence, so those two banks differ. Storage grows with your corpus rather than your vocabulary.
One vector per sentence or document, which is what retrieval runs on and what most of this page assumes.
Per-node graph vectors (graph data representations) and shared-space multimodal vectors (multimodal data fusion), where an image and its caption land in one space.

Normalise first, or your ranking quietly changes

Divide a vector by its L2 norm and you get a unit vector. Do it to both sides and the dot product is the cosine:

\hat{a} \cdot \hat{b} = \frac{a \cdot b}{\lVert a \rVert \, \lVert b \rVert} = \cos\theta

Every symbol: $a$ and $b$ are two embeddings, $\lVert a \rVert$ is the length of $a$ , $\hat{a} = a / \lVert a \rVert$ is the same direction at length 1, and $\theta$ is the angle between them.

Almost every index you'll reach for assumes you did this. Run inner-product search on unnormalised vectors and length walks into the ranking: a longer document has a longer vector, so it scores higher against everything, relevant or not. Normalise and only direction counts. Same index, same query, different order, no error.

Dimensions, bytes and the index

Width is a cost, and the arithmetic is one line: rows times dimensions times 4 bytes for float32. Ten million vectors at 1536 dimensions is $10^7 \times 1536 \times 4 = 61{,}440{,}000{,}000$ bytes, so about 61GB before the index adds its own overhead. That number is why quantisation exists. int8 keeps one byte per dimension instead of four, turning 61GB into 15GB for usually a point or two of recall. Binary packs one bit per dimension: 192 bytes a vector, 1.9GB in total, and a much bigger accuracy hit, so it comes paired with a full-precision rerank of the top few hundred candidates.

Search is the other bill. Exact nearest-neighbour search is a linear scan of every stored vector on every query, so it grows with your corpus (computational complexity). Approximate structures win that back by not looking at everything. HNSW builds a navigable graph and walks it. IVF clusters the space and searches a few cells. Both trade recall for latency, and both give you a knob that moves the trade.

So measure it. Push a few thousand queries through exact search, push the same ones through your index, and report recall@k: the share of the true top-k your index actually returned. Below roughly 0.95 at $k = 10$ , users are getting worse results than your offline numbers claim.

One version, one space, and what geometry can't say

Two vectors are comparable only when they came from the same model at the same version. Change either and every stored vector has to be recomputed, because the new space and the old one have no relationship at all. The axes aren't the same axes. Nothing errors, because the shapes still match.

Three limits worth saying out loud. No dimension has a name — 402 isn't "formality", and reading it that way is reading noise. Cosine similarity is not relevance: two documents on one subject score high whether or not either answers the question, which is why serious retrieval adds a reranker. Bias transfers — whatever the encoder learned about how words travel together is now geometry, so stereotypes come back as near neighbours.

Drift runs backwards from what you'd expect. The encoder is frozen, so a given input always yields the same vector and the stored vectors never move. Your inputs move (data drift and distribution shift). New content about something the encoder rarely saw lands in a thin region where the nearest neighbours are far away and close to arbitrary, and the symptom is a slow slide in click-through with no code change anywhere.

Worked example

A query $q = (3, 4, 0)$ and two stored documents: $a = (4, 3, 0)$ , close in direction, and $b = (0, 10, 5)$ , pointing elsewhere and longer.

Lengths first. $\lVert q \rVert = \sqrt{9 + 16} = 5$ , $\lVert a \rVert = \sqrt{16 + 9} = 5$ , $\lVert b \rVert = \sqrt{100 + 25} = 11.180$ .

Cosines. $q \cdot a = 12 + 12 = 24$ , so $24 / (5 \times 5) = 0.96$ . Then $q \cdot b = 0 + 40 + 0 = 40$ , so $40 / (5 \times 11.180) = 0.716$ . Cosine ranks $a$ first, which is right.

Now drop the normalising and rank on the raw dot product. 40 beats 24, so $b$ wins. Nothing about either document changed. $b$ is just longer, and length is what you accidentally ranked on.

Show Me the Code

1import numpy as np2
3query = np.array([3.0, 4.0, 0.0])4near = np.array([4.0, 3.0, 0.0])       # same length as the query, close in direction5long_doc = np.array([0.0, 10.0, 5.0])  # longer vector, pointing somewhere else6
7
8def unit(v: np.ndarray) -> np.ndarray:9    """L2-normalise, so a dot product against it is a cosine."""10    return v / np.linalg.norm(v)11
12print(round(float(unit(query) @ unit(near)), 3))      # -> 0.96   cosine puts this first13print(round(float(unit(query) @ unit(long_doc)), 3))  # -> 0.71614print(round(float(query @ near), 1))                  # -> 24.0   raw dot product15print(round(float(query @ long_doc), 1))              # -> 40.0   and now the long one wins16
17rows, dim = 10_000_000, 153618print(rows * dim * 4 / 1e9)                           # -> 61.44  GB as float3219print(rows * dim * 1 / 1e9)                           # -> 15.36  GB quantised to int8

One missing call to unit flips the ranking, and the search still returns ten results.

Watch Out For

Two model versions inside one index

You upgrade the embedding model, re-embed new documents as they arrive, and leave the existing rows alone because a full re-embed takes six hours of GPU time.

Nothing breaks. The dimensions match, the writes succeed, the queries return ten neighbours ranked by a number between −1 and 1. Those numbers are meaningless across the boundary: the two models put their axes in unrelated places, so a cosine of 0.91 between an old vector and a new one measures nothing. Roughly half your results are noise, and the only visible symptom is that relevance got slightly worse.

Make the model version part of the index's identity — in the collection name, not a metadata field somebody forgets to filter on. Then a model upgrade is what it actually is: build a new index, backfill every row, swap, drop the old one. Budget for that before you pick a model, because it's the recurring cost of running embeddings, not a one-off.

Inner-product search on vectors nobody normalised

The encoder returns unnormalised vectors, the store defaults to inner product, and the reflex is to write both down and move on.

Now length is in your ranking. A 4,000-word page has a bigger vector than a 200-word answer, so it scores higher against every query. Your top results skew long, generic and thin on the actual question, and the numbers coming back still look like similarities. The worked example above is the whole failure in three integers: cosine says 0.96 versus 0.716, the raw dot product says 24 versus 40, and they disagree about which document wins.

L2-normalise at write time and at query time, then use inner product or cosine — on unit vectors they're the same operation. Check one stored vector's norm before you trust anything: if it isn't 1.0, you found the bug. More on why the two metrics diverge in distance and similarity metrics.

The Quick Version

One-hot puts every pair of categories at the identical distance, so it carries no similarity at all. An embedding's geometry is the information.
This is a data problem: store, version, index, monitor. Training the encoder is somebody else's page.
L2-normalise and the dot product becomes a cosine. Skip it and document length silently enters your ranking.
Memory is rows times dimensions times 4 bytes: 10 million at 1536 dimensions is about 61GB, 15GB at int8, 1.9GB binary with a rerank to recover accuracy.
Exact search is linear; HNSW and IVF trade recall for latency. Measure recall@k against exact search, and keep it above about 0.95 at $k = 10$ .
A vector is comparable only to vectors from the identical model version. Upgrading means re-embedding everything.
No dimension has a meaning, cosine isn't relevance, and the encoder's bias is now part of the geometry.
The vectors don't drift because the encoder is frozen. Your inputs drift into thin regions of the space instead.