Learned Sparse Retrieval
SPLADE reads each document with a neural network, guesses extra vocabulary terms that fit it, and files the document under those terms too. Search stays an exact index lookup, but synonyms now match.
Why Does This Exist?
Type "car" into a parts catalog whose manual says "automobile" and classic keyword search finds nothing. That vocabulary mismatch is the wound BM25 and inverted indexes can't heal on its own: it scores only words that literally appear, so a perfect document written in different words stays invisible.
Dense retrieval with embedding models heals it by matching meaning instead of words. But it asks for new machinery: a vector index, approximate nearest-neighbor search, and real compute at query time, while giving up the exactness and the readable "why this matched" that make lexical search cheap to run. The full trade map sits in dense vs sparse vs hybrid retrieval.
Learned sparse retrieval takes a third path. A transformer reads each document, predicts which vocabulary terms fit it, including terms that never appear, and stores a weight per term. The result is still a term-to-weight list, so it runs on the same inverted index you already operate. You keep exact lookup. You gain synonym recall.
This page covers that expansion mechanism and its scoring. For the ranking math it builds on, the BM25 page comes first; for how this plugs into production beside dense vectors, hybrid search is next.
The one idea you need first. An inverted index files each word under the documents holding it, so lookup is instant, and BM25 ranks the matches by term frequency, rarity, and document length. Everything here reuses that index; only the weights filed in it change. The full treatment is in BM25 and inverted indexes, and the dry vocabulary behind it in sparse retrieval.
Think of It Like This
A parts clerk who files under every name a part goes by
You run a parts counter. A box arrives labeled "auto". A literal clerk files it under one card: auto. A customer asking for "car" leaves empty-handed.
A better clerk knows the trade. She files the same box under auto, car, vehicle, and motor, and scribbles a relevance number on each card: car gets a high number, motor a lower one. The customer asks for "car" and she pulls the box in seconds. Nothing about the filing cabinet changed. Only the cards got smarter.
Where it stops: the clerk chooses cards on purpose and never misfiles. SPLADE assigns continuous weights by pattern matching, so it sometimes files a confident card for a wrong term, and unlike the clerk it never reads the page as a whole. It votes per token and keeps the loudest vote.
How It Actually Works
SPLADE, short for Sparse Lexical and Expansion Model, starts from a transformer encoder, usually one from the BERT family, and reuses its masked-language-modeling head: the component trained to predict missing words. Instead of predicting one masked word, you ask it to score every vocabulary term against the document. Vocabulary here means the tokenizer vocabulary, about thirty thousand terms for a BERT-style model, so the output has one slot per term.
From tokens to one weight per term
For each token position in the document and each vocabulary term , the head emits a raw score (a logit: an unbounded number where a higher value means a better fit). In plain words: walks across the words you have, walks across every word you could file under, and says how strongly position votes for term .
Three operations turn votes into the filed weight for term :
- ReLU zeroes every negative vote. A term nobody voted for contributes nothing.
- compresses the survivors, so one screaming vote can't drown the rest. This step is called log-saturation.
- Max over keeps only the strongest compressed vote per term.
Terms with no positive vote land on exactly zero, and most of the thirty thousand do. What survives is a short list of terms with weights: some lifted straight from the text, some, like car for a document about autos, added by the model. That added part is learned query expansion, done once at index time instead of on every query.
The pressure that keeps it sparse
Left alone, the network would keep thousands of small weights per document and your index would swell into a slow dense one. Training therefore adds a sparsity penalty beside the usual ranking loss: for each term it averages that term's weight across the training batch to get (read it as "how much term fires on average"), then penalizes the sum of those squared averages, . A term that fires weakly everywhere costs more than it earns and gets pushed to exactly zero. Turn that penalty up and the index shrinks but recall drops; turn it down and recall climbs while the posting lists fatten. That's the central trade of the method.
Worked example
Document: "auto repair". Two token positions: for auto, for repair. Watch five vocabulary terms. The head's votes :
- auto: 3.0 from token 1, 0.5 from token 2. Compressed: , . Max: 1.39.
- repair: 0.2, 3.2. Compressed: 0.18, 1.44. Max: 1.44.
- car: 2.2, 0.8. Compressed: 1.16, 0.59. Max: 1.16. Absent from the text, present in the index.
- vehicle: 1.0, 1.6. Compressed: 0.69, 0.96. Max: 0.96. Also added.
- banana: , . ReLU zeroes both, . Filed weight 0, never stored.
All logs are natural logs, rounded to two places: , , , . Those four rounded values are the bars in the diagram above.
At query time
The query passes through the same encoder and comes out as a small weight list of its own. Say "car" expands to car 1.50, auto 1.00, vehicle 0.50. The score is a dot product over shared terms only:
score = 1.16 × 1.50 + 1.39 × 1.00 + 0.96 × 0.50 = 1.74 + 1.39 + 0.48 = 3.61.
Banana contributes nothing on either side, and repair never enters because the query gave it no weight. Plain keyword search on the same pair scores nothing at all: the query word appears nowhere in the document, so there is no shared term to score.
One practical variant expands documents only and leaves queries raw, often called SPLADE-doc. You lose query-side synonyms, but the query path needs no GPU, which is exactly the trade you want when queries arrive on hardware you don't control.
When not to use it: for exact-string hunting, part numbers, names, and error codes, expansion is a liability, not a gift. A query for "E4021" gains nothing from synonyms and can lose to a lookalike the model fancies. Keep a raw lexical signal beside it, which is where hybrid search earns its keep, or skip expansion for that field. The sibling idea that keeps a vector per token instead of a weight per term is late-interaction retrieval.
Code
The point of this snippet is the check, not the library call. It replays the worked example in plain Python: compress each vote, keep the max per term, then score the query by dot product. If your mental math matches its output, you own the mechanism.
import math
def splade_weight(votes: list[float]) -> float: """Max-pooled log-saturated weight for one vocabulary term.""" return max(math.log1p(max(0.0, v)) for v in votes)
doc_votes: dict[str, list[float]] = { "auto": [3.0, 0.5], "repair": [0.2, 3.2], "car": [2.2, 0.8], "vehicle": [1.0, 1.6], "banana": [-1.0, -0.5],}doc = {term: round(splade_weight(v), 2) for term, v in doc_votes.items()}print(doc) # -> {'auto': 1.39, 'repair': 1.44, 'car': 1.16, 'vehicle': 0.96, 'banana': 0.0}
query = {"car": 1.5, "auto": 1.0, "vehicle": 0.5}score = round(sum(doc[t] * w for t, w in query.items()), 2)print(score) # -> 3.61The dict shows the expansion plainly: car and vehicle hold real weight though the document never said them, and banana sits at 0.0, costing the index nothing. The score replays the hand computation: 1.74 + 1.39 + 0.48 = 3.61.
Watch Out For
No sparsity pressure, and your lean index turns dense
The ranking loss alone never asks for zeros. Train without the sparsity penalty, or set it near zero, and each document keeps thousands of small weights. The inverted index still works, but every posting list the junk terms touch grows long, storage climbs, and queries slow toward dense-retrieval cost with none of its quality.
Symptom: the median nonzero-term count per document climbs into the thousands while ranking metrics look fine in evaluation. Fix: log that count during training, raise the sparsity weight until the count settles near your index budget, and re-check recall after each step. Sparsity is a metric you watch, not a default you trust.
Expansion outranks the exact match
Expansion buys recall with precision's money. On exact-string queries, serial numbers, names, and error codes, an expanded lookalike can score above the document holding the literal string, and the user watches the right answer sit at rank two.
Symptom: exact queries return confident near-misses above the verbatim match. Fix: never serve expansion alone on exact fields. Keep the raw lexical score beside it and fuse the two, which is the standard hybrid search move, or route exact-pattern queries to a pure BM25 field and finish with reranking on the merged shortlist.
The Quick Version
- Learned sparse retrieval files each document under vocabulary terms with learned weights, including terms the document never says.
- Each term's weight is the loudest log-compressed vote across the document's tokens, and negative votes count as zero.
- A sparsity penalty during training forces most weights to exactly zero, so the result still runs on a normal inverted index.
- The query expands the same way, and the score is a dot product over the terms both sides kept.
- A query for "car" finding a document that only says "auto" is the whole point; "E4021" matching a lookalike instead of the exact code is the failure to guard.
- Expansion alone is not a stack: pair it with raw lexical scoring for exact queries and a reranker on top.