Embedding Fine-Tuning
A public embedding model ranks words the way the web uses them. Fine-tuning re-trains it on your own query and document pairs, so your meaning of a term is the one that wins.
Why Does This Exist?
A public embedding model learns what words mean from the public web. That works until your words don't mean what the web thinks. In a hospital, MI means myocardial infarction, but the model has also seen countless pages where MI means Michigan, so the query "MI treatment protocol" can rank a page about Michigan above the cardiology protocol. Your semantic search looks confident and returns the wrong document, and everything downstream answers from it.
The fix is to keep the model and change what it counts as close. Embedding models are trained so related texts land near each other, and this page adapts that training to your vocabulary using your own labeled pairs. You will see the exact data format, the loss that does the pulling and pushing, and a worked ranking you can check by hand.
The one idea you need first. An embedding is a list of numbers standing in for a piece of text, and closeness in that number space means relatedness. Embeddings covers how that space gets built, and similarity metrics covers how closeness gets scored. Here you need only one scoring rule: cosine similarity, the angle between two vectors, running from (opposite) to (identical direction).
Think of It Like This
A library rearranged for its regulars
A new librarian inherits a sensible shelf order built for the average reader. Then the regulars arrive, and they all ask in local shorthand: "the MI file" always means the cardiology binder, never the Michigan travel shelf. She doesn't rebuild the library. She moves the requested binder to the front of the right bay, and she moves the travel book that kept disappointing people to the back. Same books, new distances, tuned to the people who actually walk in.
That's what you're doing to the vector space: promoting what your readers reach for, demoting the lookalike that wastes their time.
Where it stops: books sit still once shelved, but vectors never do. Every training step shifts the whole space at once, so while you fix your corner you can quietly wreck the general ordering. That's why the page ends with a held-out test, not with training loss.
How It Actually Works
Training pairs say what "close" means here
Raw text is not enough. You must show the model decisions, in one of two formats. A positive pair is a query plus its matching document: ("MI treatment protocol", "Administer aspirin for suspected myocardial infarction."). A triplet adds a hard negative, a document that looks relevant but is wrong: the same query, the aspirin passage, plus "MI ranked 12th in national parks near Michigan lakes." Triplets teach faster because each example names both directions: toward this, away from that.
Quality beats quantity in a specific sense. Random negatives (cardiology versus cooking) teach almost nothing since any space already separates them. A hard negative shares words with the query yet answers nothing, which forces the fine distinctions your retrieval actually gets wrong. Mining them usually means taking your current system's top results and having a person mark which ones disappoint.
One loss pulls the match and pushes the rest
Training is contrastive: each step drags the query vector toward its match and shoves it away from the rivals. The workhorse for pair data is Multiple Negatives Ranking Loss. For a query with its match inside a batch of documents :
In plain words: is cosine similarity, (tau) is the temperature, a small positive number that sharpens the comparison, and is the batch size. Every other document in the batch serves as a free negative, which is why bigger batches train better and cost more memory. When you have explicit triplets, the margin form says it more directly, with the hard negative and the margin, the gap you demand between match and rival:
If the match already beats the rival by more than , the loss is zero and the example teaches nothing further. That cutoff is a feature: training spends its effort where the ranking is still wrong.
Worked example: MI treatment protocol
Take a query , its true match , and a Michigan lookalike . Cosine similarity is the dot product divided by both lengths. Before tuning:
The lookalike wins 0.976 to 0.707, so the search returns Michigan. That's the failure the diagram draws on its left panel. With temperature and these two documents as the batch, the ranking loss is about 5.39, and the margin loss at is . Both numbers agree: the ranking is wrong and the model should move.
After tuning on your pairs, the vectors have rotated: sits near the query while points elsewhere:
Now the match wins 0.980 to 0.301, the ranking loss has collapsed to practically zero, and the margin loss is . Nothing left to learn from this example, which is exactly what a loss of zero should mean.
Two honest limits before you start. First, the pairs must mirror your live traffic; if real queries drift to new shorthand, the tuned space goes stale and you tune again. Second, scale matters: hundreds of representative pairs reshape a space, while a dozen pairs just memorize themselves. If you cannot collect the pairs, don't tune yet. A rewritten query or a reranker usually beats tuning on data you don't have, and when to fine-tune walks through that call.
Code
The point of this snippet is the check, not the formula: recompute the four cosines from the worked example and confirm the ranking flips.
import math
def cosine(a: list[float], b: list[float]) -> float: dot = sum(x * y for x, y in zip(a, b)) na = math.sqrt(sum(x * x for x in a)) nb = math.sqrt(sum(y * y for y in b)) return dot / (na * nb)
q = [1.0, 0.0]p_before = [0.7, 0.7]n_before = [0.9, 0.2]p_after = [0.98, 0.2]n_after = [0.3, 0.95]
print(round(cosine(q, p_before), 3)) # -> 0.707print(round(cosine(q, n_before), 3)) # -> 0.976print(cosine(q, n_before) > cosine(q, p_before)) # -> Trueprint(round(cosine(q, p_after), 3)) # -> 0.98print(round(cosine(q, n_after), 3)) # -> 0.301print(cosine(q, p_after) > cosine(q, n_after)) # -> TrueThe two True lines are the whole story: wrong order before, right order after, on the exact numbers the diagram draws.
Watch Out For
Tuning on a thimbleful of pairs
Fifty lovingly labeled pairs, twenty epochs, training recall near perfect, and live search gets worse. That's the standard shape of this failure: the model memorized your examples and blurred everything else, including plain language it used to handle fine. Training loss said "done" while held-out queries quietly rotted.
Judge the tuned model on queries it never trained on, plus a small set of general queries as a tripwire. Train for one to three epochs at a small learning rate, mix in some general pairs so the space stays anchored, and stop when held-out ranking stalls. Prove the gain the way RAG evaluation describes, never with the loss curve alone.
Negatives so easy the loss learns nothing
The loss falls smoothly, the vectors barely move, and live ranking doesn't budge. With tiny batches of random pairs, every in-batch negative is already far away, so the denominator of the loss is dominated by the match and the gradient is near zero. The optimizer is being told everything is already fine.
Mine hard negatives: take your current top results, keep the ones a person marks wrong, and train triplets on those. Grow the batch until each query sees real rivals, or switch to the margin form where each example names its rival explicitly. If loss drops without rank movement, suspect the negatives before you suspect the model.
The Quick Version
- Fine-tuning an embedding model re-trains it on your own query and document pairs, so domain terms rank the way you use them.
- Training data is structure, not raw text: query plus matching document, ideally with a hard negative that looks relevant but is not.
- The standard loss is Multiple Negatives Ranking Loss: each query is pulled toward its match and pushed from every other document in the batch, sharpened by temperature tau.
- On the worked example the lookalike beats the match 0.976 to 0.707 before tuning and loses 0.301 to 0.980 after, with the loss falling from about 5.39 to nearly zero.
- Small pair sets overfit fast, so judge the tuned model on held-out queries rather than on training loss.
- Skip fine-tuning when you can't collect representative pairs, since a rewritten query or a reranker usually wins there.