Skip to content
AI360XpertBeta

Sentence Embeddings

One dense vector per whole sentence, trained so that paraphrases land close together and unrelated sentences land far apart. Meaning you can compare with a ruler.

Averaged word vectors give swapped sentences the same vector, while a trained sentence encoder puts paraphrases close and strangers far
Averaged word vectors give swapped sentences the same vector, while a trained sentence encoder puts paraphrases close and strangers far

Why Does This Exist?

A search box rarely matches word for word. You type "how do I reset my password" and the help page says "credential recovery procedure". No words overlap, yet the meanings line up. Keyword search fails here every day, and it fails because it compares strings instead of meanings.

Word2Vec taught us to compare single words as vectors. Sentences need the same trick, and the lazy version breaks immediately. Average the word vectors of "dog bites man" and you get exactly the same vector as "man bites dog". Same words, same average, opposite meanings. Word order, syntax, and who did what to whom all wash out.

Sentence embeddings fix this by training a model to output one vector for the entire sentence directly, with order intact. Two paraphrases land near each other no matter how different their words are. That single property powers semantic search, duplicate detection, and clustering of text at scale.

The one idea this page stands on is the word vector: words as points where distance means similarity. This page lifts that idea from words to whole sentences. If word vectors are new, start with Word2Vec, then come back.

Think of It Like This

Averaging paint against placing paintings in a gallery

Imagine judging a painting of a sunset by averaging every colour on the canvas. A sunset scene and a desert scene could average to the same muddy brown. Same average, completely different paintings. Every compositional choice, where the river runs and what stands in front of what, is gone.

Sentence embeddings skip the averaging. They walk the whole gallery wall and hang each painting next to its neighbours: sunsets with sunsets, portraits with portraits, whatever the palette. Two sunset paintings hang together even if one uses orange and the other pink.

The analogy stops at the front door of the gallery. Paintings hang in two dimensions a visitor can see. Sentence vectors live in hundreds of dimensions no one can picture, and closeness there is a computed number, not a glance. The gallery explains the organisation. Only the arithmetic explains the placement.

How It Actually Works

Two copies, one judge

The modern standard is the Siamese setup behind Sentence-BERT. Take one encoder, usually a pretrained transformer, and run two sentences through the same weights independently. Pool each output down to a single vector, often 384 numbers for the compact public models. Then compare the pair with cosine similarity:

cos⁡(u,v)=u⋅v∥u∥ ∥v∥\cos(u, v) = \frac{u \cdot v}{\lVert u \rVert \, \lVert v \rVert}

In plain words: uu and vv are the two sentence vectors, u⋅vu \cdot v is their dot product, and each ∥⋅∥\lVert \cdot \rVert is the vector's length, the same norm idea as ruler distance. Dividing by both lengths keeps the score between -1 and 1 no matter how long the vectors are. Near 1 means same direction, same meaning. Near 0 means unrelated.

Training on pairs, not labels

Nobody labels vectors directly. Training feeds sentence pairs already marked as paraphrase or not, or ranked by entailment, and nudges the weights so matching pairs score near 1 and mismatched pairs score near 0 or below. That is contrastive training applied to sentences, and the mechanism has its own page in contrastive learning. At search time both sides get encoded once, and every comparison is one cheap cosine between frozen vectors.

Worked example

Take two toy sentence vectors, a=[0.9,0.1]a = [0.9, 0.1] and b=[0.8,0.2]b = [0.8, 0.2], standing in for two paraphrases. Their dot product is 0.9×0.8+0.1×0.2=0.740.9 \times 0.8 + 0.1 \times 0.2 = 0.74. Their lengths are ∣a∣=0.82≈0.906|a| = \sqrt{0.82} \approx 0.906 and ∣b∣=0.68≈0.825|b| = \sqrt{0.68} \approx 0.825. So cos⁡(a,b)=0.74/(0.906×0.825)≈0.99\cos(a, b) = 0.74 / (0.906 \times 0.825) \approx 0.99. Near identical direction, near identical meaning.

Against c=[0.1,0.9]c = [0.1, 0.9], an unrelated sentence, the dot product with aa is 0.09+0.09=0.180.09 + 0.09 = 0.18, and cos⁡(a,c)=0.18/0.82≈0.22\cos(a, c) = 0.18 / 0.82 \approx 0.22. Same arithmetic, near zero score. That gap between 0.99 and 0.22 is what search ranks on.

Watch Out For

Searching short queries with a symmetric model

Standard sentence models train on sentence-to-sentence pairs, so both sides look alike. A search query is not a sentence: three words against a three hundred word document. The symmetric model never practised that matchup, and results go soft. The symptom is specific: paraphrase tests look great while real query logs disappoint.

Reach for a model trained asymmetrically, with short queries on one side and long passages on the other, the way retrieval benchmarks pair them. Symmetric similarity and asymmetric search are different tasks sharing one cosine.

Treating a cosine score as absolute

A threshold of 0.8 that separates duplicates cleanly on one model can be noise on another. Embedding spaces differ in how spread out they are, so the same number means different things per model. The symptom is a threshold tuned on a demo that collapses on your data.

Calibrate per model on your own labelled pairs before trusting any cutoff. Pick the threshold from a precision curve you measured, never from a number that worked elsewhere.

The Quick Version

  • Averaging word vectors destroys order: "dog bites man" and "man bites dog" average identically.
  • Sentence embeddings encode the whole sentence into one vector with order intact, and paraphrases land close together.
  • The Siamese setup runs both sentences through shared weights and trains their cosine similarity toward 1 for matches.
  • At search time every document is encoded once, so each query costs one cheap cosine per candidate.
  • Symmetric models compare sentences well and search queries poorly. Match the model to the matchup.
  • Cosine scores are relative per model. Calibrate thresholds on your own pairs.