Vector Databases

An ANN index finds close vectors fast, but a product needs more: only your tenant's current chunks, durable across restarts, with inserts and deletes that actually stick.

Tenant filtering plus ANN ranking returns d1 at 1.00 and d2 at 0.95 for tenant_a while tenant_b d3 at 0.60 stays excluded

Why Does This Exist?

You ship a support bot over two million help chunks and the demo feels instant. Then the second customer signs up and asks the question you dread: can anyone see anyone else's docs? Then support asks for deletion of last year's prices, product asks to search only current tenant docs from the last release, and your server restarts overnight and forgets the whole index. A bare nearest neighbor library answers none of that. It holds a fixed array of vectors in memory and returns the closest few. That is all it promises.

A vector database is the system around that one fast operation. It stores each vector with its metadata, enforces who may see what on every query, accepts inserts and deletes while serving traffic, and brings the index back after a crash. You pay for that wrapper in storage and moving parts. You buy the right to run one shared store for many tenants without leaking data or rebuilding from scratch each morning.

This page covers the wrapper: filtering, tenancy, writes, and persistence. The fast operation underneath lives in approximate nearest neighbor search, the most common graph index inside it lives in HNSW, and the pipeline that consumes the results lives in RAG architecture. Read those for depth. Here you get the four jobs, one worked ranking, and the two failures teams hit first.

One idea you need first, kept short: an embedding turns text into a list of numbers, and closeness is cosine similarity, the dot product divided by both lengths. For $a = [2, 1]$ and $b = [1, 3]$ the dot product $a \cdot b$ is $2 \times 1 + 1 \times 3 = 5$ , each length is the square root of the sum of squares, and the cosine is that $5$ divided by both lengths. The full treatment with the worked $0.7071$ example is in embeddings. Everything below assumes you can picture points in space with a score between them.

Think of It Like This

An apartment mailroom with locked boxes

Your building receives hundreds of parcels a day and the mailroom shelves them by size so clerks find similar boxes fast. That shelf layout is the ANN index: a clever arrangement that makes the common lookup quick.

But nobody lets tenants wander the shelves. Each apartment has a locked box, every parcel carries a name and a date, the desk keeps a logbook of what arrived and what was picked up, and the room itself has a concrete floor and a spare key off site so a burst pipe does not erase the log. Filtering is the name check, tenancy is the lock, persistence is the floor and the spare key, and inserts and deletes are the daily intake and pickup. Swapping the shelf layout does not change any of that furniture.

Where it stops: parcels sit where clerks put them and stay put. Vectors get their positions from a model, and upgrading that model moves every point at once. The mailroom never has to re-shelve the whole building because the ruler changed. You do.

How It Actually Works

An ANN index answers one question fast: given a query vector, which stored vectors score highest. A vector database decides which vectors may compete, records the answer durably, and keeps that promise while data changes. Four jobs, one ranking.

The four jobs around the index

Call the query vector $q$ : the $d$ numbers your encoder produced for the query, where $d$ is the embedding dimension. Call the stored vectors $x_1$ through $x_n$ , where $n$ is the collection size. Pick a score $s(q, x)$ where larger means closer, usually cosine similarity. The index returns the top $k$ , where $k$ is how many neighbors you asked for.

The database wraps that call:

Filtered search. Each vector carries structured fields such as tenant id, date, or source. A predicate keeps only matching rows before ranking.
Tenant isolation. One tenant's vectors never enter another tenant's result, enforced as a mandatory predicate rather than a suggestion your code might skip.
Live writes. Inserts add vectors and metadata, deletes mark rows gone even when the underlying index reclaims space later.
Persistence. Vectors, metadata, and enough index state survive a restart through disk writes plus replication, so recovery does not mean re-embedding millions of chunks.

The trade is concrete. You carry a second store for metadata, a write path with acknowledgments, and background compaction. In return many tenants share one system safely. Skip the wrapper when none of that applies: a few thousand vectors, one tenant, an ephemeral demo you rebuild on launch. Then a fixed in-memory scan is simpler and honest.

Filters decide who may rank

Write the predicate first and the similarity second. Pre-filtering keeps only rows matching the predicate and ranks those, so $k$ results arrive whenever $k$ eligible rows exist. Post-filtering ranks first and discards mismatches after, so a restrictive predicate can return one row when you asked for five. Neither is free. Pre-filtering on a highly selective predicate can force a graph walk to wander far before it collects enough matches. Measure both on your own queries when the predicate removes most rows.

Tenant filtering deserves its own sentence because the failure mode is a privacy incident, not a relevance dip. The tenant predicate belongs at the API layer where no query path can omit it, and the test suite proves it with two tenants whose chunks sit near each other in vector space. Anything weaker is an optional filter waiting for one missed call site.

Worked example with four chunks

Query $q = [3, 1]$ , so $\lVert q \rVert = \sqrt{9 + 1} = \sqrt{10} \approx 3.1623$ . Here $\lVert q \rVert$ is the length, the square root of the sum of squares. Four stored chunks use two dimensions so you can check each step:

$d_1 = [3, 1]$ , tenant_a, current. Dot $9 + 1 = 10$ , length $\sqrt{10}$ , so $s = 10 / 10 = 1.00$ .
$d_2 = [3, 0]$ , tenant_a, current. Dot $9 + 0 = 9$ , length $3$ , so $s = 9 / (3 \times 3.1623) \approx 0.9487$ .
$d_3 = [1, 3]$ , tenant_b, current. Dot $3 + 3 = 6$ , length $\sqrt{10}$ , so $s = 6 / 10 = 0.60$ .
$d_4 = [0, 2]$ , tenant_a, archived. Dot $0 + 2 = 2$ , length $2$ , so $s = 2 / (2 \times 3.1623) \approx 0.3162$ .

Now apply the predicate tenant_a and current. Eligible: $d_1$ and $d_2$ . Excluded: $d_3$ by tenant, $d_4$ by status. Rank what remains with $k = 2$ : $d_1$ at $1.00$ first, $d_2$ at $0.95$ second. Notice what the filter did. Globally $d_3$ at $0.60$ outscores $d_4$ at $0.32$ , yet neither is returned. Filters decide who may rank. Similarity decides the order among survivors. Those are the numbers drawn in the diagram above and printed by the snippet below.

Pair this with keyword search when exact tokens matter. Vectors blur rare codes and names, so an error string can drown among paraphrases. That pairing is hybrid search, and the second scoring pass over the shortlist is reranking.

Writes, deletes, and surviving a restart

Reads get the attention and writes cause the outages. An insert lands in a durable log before it becomes searchable, a delete marks a tombstone the filter honors immediately while the index reclaims the slot in the background, and a restart replays the log instead of re-embedding the corpus. That is why the same embedding model version must produce every vector in one index: each model builds its own coordinate system, and a distance across two systems measures nothing. Upgrading the model means re-embedding all chunks, and splitting long docs into passage-sized pieces first is the subject of chunking strategies.

The assumption underneath is that your filter predicates are selective enough to be worth enforcing and your update rate fits background compaction. When deletes outpace compaction or one tenant dwarfs the rest, latency spreads and recall wobbles until the store catches up. That is the moment teams discover the index was never the system.

Code

The point of this snippet is filter then rank on the worked fixture: two predicates in, one ordered shortlist out.

1import math2
3
4def cosine(a: list[float], b: list[float]) -> float:5    dot = sum(x * y for x, y in zip(a, b))6    na = math.sqrt(sum(x * x for x in a))7    nb = math.sqrt(sum(y * y for y in b))8    return dot / (na * nb)9
10
11def top_k(12    query: list[float],13    docs: dict[str, tuple[list[float], str, str]],14    k: int,15    tenant: str,16) -> list[tuple[str, float]]:17    scored: list[tuple[str, float]] = []18    for doc_id, (vec, doc_tenant, status) in docs.items():19        if doc_tenant != tenant or status != "current":20            continue21        scored.append((doc_id, cosine(query, vec)))22    scored.sort(key=lambda item: item[1], reverse=True)23    return scored[:k]24
25
26q = [3.0, 1.0]27docs = {28    "d1": ([3.0, 1.0], "tenant_a", "current"),29    "d2": ([3.0, 0.0], "tenant_a", "current"),30    "d3": ([1.0, 3.0], "tenant_b", "current"),31    "d4": ([0.0, 2.0], "tenant_a", "archived"),32}33
34ranked = top_k(q, docs, k=2, tenant="tenant_a")35print([doc_id for doc_id, _ in ranked])  # -> ['d1', 'd2']36print(round(ranked[0][1], 4))  # -> 1.037print(round(ranked[1][1], 4))  # -> 0.9487

Tenant_a keeps $d_1$ and $d_2$ in that order. Tenant_b $d_3$ never competes despite its $0.60$ , and archived $d_4$ stays out despite belonging to the right tenant.

Watch Out For

Post-filtering a short ANN shortlist

You ask for five, the index returns its five closest, four belong to other tenants, and your user sees one result with no error. It looks like a thin corpus. It is a shortlist filtered too late.

The symptom is pages that thin out exactly where the predicate bites: new tenants, narrow date ranges, rare tags. Fix it by filtering before ranking when your store supports it, or by over-fetching by a wide margin and then filtering when it does not. Either way prove it: run a query set with your tightest real predicates and count returned rows alongside recall. If the count sags under restrictive filters, the ordering step is fine and the placement of the filter is wrong.

One code path forgets the tenant predicate

Search has three callers and two pass tenant_a correctly. The third, a debug endpoint or an export job, passes no tenant at all and the store helpfully searches everything. Eval never catches it because eval uses the correct path.

The symptom arrives as a report, not a metric: a user quotes a chunk they should never have seen. Treat a missing tenant as a rejected query, never as a global search. Enforce the predicate at the API boundary, add an integration test with two tenants whose vectors sit close together, and delete any unscoped search path rather than documenting it.

The Quick Version

A vector database wraps an ANN index with tenant filters, metadata predicates, live writes, and persistence across restarts.
Filters decide who may rank and similarity decides the order: tenant_a keeps d1 at 1.00 and d2 at 0.95 while tenant_b d3 at 0.60 stays excluded.
Pre-filtering guarantees k results when k eligible rows exist, while post-filtering can return fewer after discards.
Tenant isolation is a mandatory predicate enforced at the API layer, not an optional argument each caller must remember.
Skip the database for a small single tenant ephemeral store, and pair vectors with keyword search when exact codes must match.
Every vector in one index must come from the same embedding model version, or every distance in it is noise.