Sentence Embeddings

Why Does This Exist?

Search rarely matches word for word. You type how do I reset my password and the help page says credential recovery procedure. No words overlap, yet the meanings line up. Keyword search fails there every day because it compares strings instead of meanings.

Word embeddings taught us to compare single words as vectors. Sentences need the same trick, and the lazy version breaks at once. Average the word vectors of dog bites man and you get exactly the same vector as man bites dog. Same words, same average, opposite meanings. Order and who did what wash out.

Sentence embeddings fix this by encoding the whole sentence into one vector with order intact. Paraphrases land near each other no matter how different their words are. That single property powers semantic search, duplicate detection, and clustering at scale. The encoders that produce them are trained as language models first, then tuned for similarity, and the top use is retrieving passages for question answering.

The one idea you need first

A word vector places one word at a point where distance means similarity. A sentence vector lifts that idea to a whole sentence: one point per sentence, compared with cosine similarity. Cosine divides the dot product by both lengths, so it reads direction instead of size. If word vectors are new, start with word embeddings, then come back.

Think of It Like This

How It Actually Works

Two copies plus one judge

The modern standard is the Siamese setup behind Sentence-BERT. Take one encoder, usually a pretrained transformer, and run two sentences through the same weights independently. Pool each output down to one vector, often 384 numbers for the compact public models. Then compare the pair with cosine similarity:

In plain words: $u$ and $v$ are the two sentence vectors, $u \cdot v$ is their dot product, and each $\lVert \cdot \rVert$ is the vector length, the same norm idea as ruler distance. Dividing by both lengths keeps the score between minus 1 and 1 no matter how long the vectors are. Near 1 means same direction and same meaning. Near 0 means unrelated.

Training on pairs not labels

Nobody labels vectors directly. Training feeds pairs already marked as paraphrase or not and nudges the weights so matches score near 1 and mismatches score near 0 or below. Pooling is how one vector emerges from many token vectors: mean pooling averages contextualized token outputs, while CLS pooling takes the special classification token trained to summarize. At search time both sides get encoded once, and every comparison is one cheap cosine between frozen vectors.

Worked example

Take toy sentence vectors $a = [0.9, 0.1]$ and $b = [0.8, 0.2]$ for two paraphrases. Their dot product is $0.9 \times 0.8 + 0.1 \times 0.2 = 0.74$. Their lengths are $\lvert a \rvert = \sqrt{0.82} \approx 0.906$ and $\lvert b \rvert = \sqrt{0.68} \approx 0.825$. So $\cos(a, b) = 0.74 / (0.906 \times 0.825) \approx 0.991$. Near identical direction, near identical meaning.

Against $c = [0.1, 0.9]$ for an unrelated sentence, the dot with $a$ is $0.09 + 0.09 = 0.18$, and $\cos(a, c) = 0.18 / 0.82 \approx 0.220$. Same arithmetic, near zero score. That gap between 0.991 and 0.220 is what search ranks on. Averaged word vectors cannot show it because swapped sentences average identically.

Code

This snippet replays the cosine gap so you can check that paraphrases score near 1 while strangers sit near 0.2.

1import math2from typing import List3
4
5def cosine(u: List[float], v: List[float]) -> float:6    dot = sum(x * y for x, y in zip(u, v))  # alignment of directions7    nu = math.sqrt(sum(x * x for x in u))  # length of u8    nv = math.sqrt(sum(y * y for y in v))  # length of v9    return dot / (nu * nv)10
11
12a = [0.9, 0.1]13b = [0.8, 0.2]14c = [0.1, 0.9]15print(f"paraphrase: {cosine(a, b):.3f}")16# -> paraphrase: 0.99117print(f"stranger:   {cosine(a, c):.3f}")18# -> stranger:   0.220

Both numbers match the hand division above. Drop in real 384 number vectors from any encoder and the same function ranks your candidates.

Watch Out For

The Quick Version

• Averaging word vectors destroys order: dog bites man and man bites dog average identically.
• Sentence embeddings encode the whole sentence into one vector with order intact, and paraphrases land close together.
• The Siamese setup runs both sentences through shared weights and trains their cosine toward 1 for matches.
• Toy vectors give 0.991 for paraphrases and 0.220 for strangers, and search ranks on that gap.
• At search time every document is encoded once, so each query costs one cheap cosine per candidate.
• Symmetric models compare sentences well and search queries poorly, and cosine cutoffs do not transfer across models.

What to Read Next

• Word Embeddings
• Language Models
• Question Answering