Sentence Embeddings
Sentence embeddings map entire sentences or paragraphs into dense vector spaces, capturing high-level semantic meaning for tasks like semantic search and clustering.
Why Does This Exist?
Word embeddings successfully mapped individual words into dense vector spaces, allowing models to understand semantic relationships like similarity and analogy. However, language is rarely processed one word at a time. Humans communicate in sentences and paragraphs, where the overall meaning is highly dependent on word order, syntax, and context.
Historically, practitioners tried to create sentence representations by simply taking the mathematical average of all the word vectors in a sentence. While computationally cheap, this "bag-of-words" averaging approach destroys all structural information. The sentences "The dog chased the cat" and "The cat chased the dog" contain the exact same words and thus yield the exact same average vector, despite having opposite meanings.
Furthermore, early transformer models like standard BERT were highly accurate at comparing two sentences, but they required feeding both sentences simultaneously into the network to compute an attention-based similarity score. If you wanted to find the most similar sentence in a database of one million documents, you had to run the heavy transformer model one million times. This was computationally prohibitive for large-scale semantic search.
Sentence embeddings solve this by encoding an entire sentence or paragraph into a single, fixed-size dense vector. This allows the semantic meaning of the entire sequence to be captured structurally, and crucially, it allows millions of sentence vectors to be pre-computed, indexed, and compared near-instantly using simple geometric operations like cosine similarity.
Think of It Like This
Summarizing a movie plot
Imagine trying to compare two movies to see if they are similar. If you only use "word embeddings," you are essentially just comparing their cast lists. If both movies feature the exact same actors, a word-averaging approach assumes they are the identical film, even if one is a comedy and the other is a tragedy.
Sentence embeddings are like writing a concise, one-paragraph summary of the movie's actual plot, themes, and tone, and then mapping that summary to a specific genre coordinate system. Once you have these high-level plot coordinates, you don't need to re-watch the movies to compare them; you simply measure the distance between their coordinates. If someone asks for a "sci-fi thriller about time travel," you instantly look for the closest plot coordinates in your database, bypassing the raw cast lists entirely.
How It Actually Works
Modern sentence embeddings are predominantly generated using specialized transformer architectures, the most famous being Sentence-BERT (SBERT). Here is the technical mechanism behind generating and using these sequence-level vectors.
1. The Siamese Network Architecture
To train a model to produce high-quality sentence embeddings, architectures often use a "Siamese" or dual-encoder network. During training, two identical transformer networks (sharing the exact same weights) process two different sentences simultaneously.
2. Pooling Strategies
A standard transformer outputs a vector for every single token (word piece) in the sentence. To compress this into a single sentence-level vector, a "pooling" operation is applied to the final output layer. The most common strategies are:
- MEAN pooling: Averaging the token embeddings, but doing so after the transformer's self-attention mechanism has heavily contextualized them.
- CLS pooling: Taking the output vector corresponding to a special
[CLS](classification) token prepended to the input sequence, which is trained to act as an aggregate representation.
3. Contrastive Learning and Fine-tuning
The Siamese network is trained on pairs of sentences using contrastive learning. If two sentences are semantically similar (e.g., "The weather is rainy" and "It is pouring outside"), the loss function penalizes the network if their pooled sentence vectors are far apart. If they are dissimilar or contradictory, the network is penalized if they are too close. By minimizing this loss over massive datasets of sentence pairs, the shared transformer weights learn to map conceptually similar sequences to adjacent regions in the vector space.
4. Vector Search and Indexing
Once trained, the model is detached from the Siamese setup. A single transformer is used to pre-compute the sentence embedding for every document in a database. These vectors (often 384 or 768 dimensions) are stored in a specialized vector database. When a user issues a search query, the query is passed through the same model to generate a query vector. The database then performs a highly optimized Nearest Neighbor search to retrieve the pre-computed sentence vectors with the highest cosine similarity to the query vector.
Code
This is a minimal demonstration using the popular sentence-transformers library to generate embeddings for entire sentences and rapidly compute their semantic similarity.
from sentence_transformers import SentenceTransformerfrom sklearn.metrics.pairwise import cosine_similarityimport numpy as np
# Load a pre-trained, lightweight sentence transformer modelmodel = SentenceTransformer('all-MiniLM-L6-v2')
# Define a set of sentences to encodesentences = [ "The chef prepared a delicious meal.", "A fantastic dinner was cooked by the culinary expert.", "The dog barked loudly at the mailman.", "I need to wash my car this weekend."]
# Generate dense vector embeddings for all sentences (returns a numpy array)# This executes the transformer pass and the pooling operationembeddings = model.encode(sentences)print(f"Shape of embeddings: {embeddings.shape}")# -> Shape of embeddings: (4, 384)
# Calculate cosine similarity between the first sentence and all othersquery_vector = embeddings[0].reshape(1, -1)similarities = cosine_similarity(query_vector, embeddings)[0]
for i, score in enumerate(similarities): print(f"Similarity to sentence {i}: {score:.3f}")
# -> Similarity to sentence 0: 1.000 (Exact match)# -> Similarity to sentence 1: 0.812 (High semantic similarity despite different words)# -> Similarity to sentence 2: 0.051 (Unrelated)# -> Similarity to sentence 3: 0.015 (Unrelated)Watch Out For
Asymmetric vs. Symmetric Search
When building semantic search systems, be aware of the difference between symmetric and asymmetric search. In symmetric search, the query and the target documents are roughly the same length and format (e.g., finding similar FAQs). In asymmetric search, a short query (e.g., "how to fix a tire") is used to search through long documents (e.g., entire manuals). You must choose a pre-trained sentence embedding model explicitly tuned for your specific search type, as a symmetric model will perform poorly on asymmetric tasks.
Context Window Limits
Sentence embedding models, being based on transformers, have strict maximum sequence lengths (often 512 tokens). If you pass a document longer than this limit, the model will silently truncate the text, generating an embedding based only on the first few paragraphs. To embed long documents, you must chunk the text into smaller segments, embed each chunk independently, and search across the chunks.
The Quick Version
- Sentence embeddings compress entire phrases, sentences, or paragraphs into a single dense vector, capturing structural and contextual meaning.
- They overcome the limitations of simple word-averaging, which ignores word order, and standard cross-encoder transformers, which are too slow for large-scale retrieval.
- Models like Sentence-BERT use Siamese network architectures and contrastive learning to train the underlying transformer to output semantically meaningful vectors.
- In production, millions of vectors are pre-computed, indexed in vector databases, and retrieved near-instantly using cosine similarity.
- They are the foundational technology behind modern semantic search, Retrieval-Augmented Generation (RAG), and zero-shot clustering.