Skip to content
AI360XpertBeta

Semantic Chunking

Split a document where its meaning shifts, not every 500 characters, so each chunk you embed and retrieve holds one whole idea.

Sentence pair similarities of 0.98, 0.65 and 0.99 split the text where the score cliffs below the 0.80 threshold, so each chunk holds one topic
Sentence pair similarities of 0.98, 0.65 and 0.99 split the text where the score cliffs below the 0.80 threshold, so each chunk holds one topic

Why Does This Exist?

A retrieval system never embeds a whole manual as one vector. It splits each document into chunks first, then embeds every chunk on its own, because one vector stretched over ten topics matches no query well. The glossary form of the term is chunking.

The cheapest way to split is fixed-size: cut every few hundred tokens wherever the counter lands. It is fast, predictable, and blind. Sooner or later a boundary lands mid-explanation, and the two halves get embedded separately. Your query then matches one half, the model reads a fragment, and the answer comes out thin. You will recognise the symptom: the right document exists, yet the retrieved passage never contains the full thought.

Semantic chunking cuts where the meaning shifts instead. Sentences that talk about the same thing stay together; a new chunk starts only when the topic genuinely turns. Each chunk then holds one whole idea, which is exactly what an embedding represents well.

The one idea underneath: an embedding is a list of numbers standing in for a piece of text, and closeness in that number space means relatedness. The ruler used here is cosine similarity, the angle between two vectors, scored in similarity metrics. If those two pages feel rusty, skim them first, then come back. Everything below assumes you can picture sentences as points with a measurable gap between neighbours.

This page covers the meaning-based cut itself: the score, the threshold, and the safety cap. The full menu of strategies lives in chunking strategies, and the pipeline that consumes the chunks lives in RAG architecture.

Think of It Like This

Cutting a home movie into clips

You have three hours of unsorted family footage and you want watchable clips. One option is cutting every ten minutes on the dot. Fast, even, and wrong: the birthday song gets sliced in half, and blowing out the candles lands in a different clip from making the wish.

The better option is cutting when the scene changes. Birthday party in one clip, backyard football in the next. Clips come out uneven, and nobody minds, because each one holds a single event you can actually enjoy.

Sentences work the same way. Fixed-size chunking is the ten-minute timer. Semantic chunking watches for the scene change and cuts there.

Where it stops: a movie has hard visual cuts, and your eyes catch them for free. Prose drifts instead of cutting, so "the scene changed" is a slope, not a snap, and the threshold is your judgment call. Embeddings can also be fooled by things your eyes never would be, like a negation flipping a sentence's meaning while keeping its words. The score measures angle, not understanding.

How It Actually Works

Call the document's sentences s1,s2,…,sns_1, s_2, \ldots, s_n, in order. Each sentence sis_i is turned into an embedding eie_i, which is just the sentence written as a list of numbers. Between every pair of neighbours compute one number, the similarity sim(ei,ei+1)\mathrm{sim}(e_i, e_{i+1}), which says how much meaning the two adjacent sentences share. Pick a cutoff τ\tau, the Greek letter tau, called the threshold. Wherever the similarity dips below τ\tau, draw a chunk boundary. That is the whole method: high score, keep grouping; low score, cut.

From sentences to scores

The standard score is cosine similarity. For two vectors aa and bb:

sim(a,b)=a⋅b∥a∥ ∥b∥\mathrm{sim}(a, b) = \frac{a \cdot b}{\lVert a \rVert \, \lVert b \rVert}

In plain words: a⋅ba \cdot b is the dot product, the sum of the pairwise products of the entries. ∥a∥\lVert a \rVert is the length of aa, the square root of the sum of its squared entries, and ∥b∥\lVert b \rVert is the same for bb. Dividing the alignment by the two lengths means you compare directions, not sizes, so a long sentence and a short one can still score near 11 when they point the same way. The score is 11 for identical direction, 00 for unrelated, and negative past a right angle.

One practical refinement: single sentences are noisy. A three-word transition like "The result:" matches nothing and can fake a topic shift. Production setups embed small rolling windows, say three sentences at a time, and score window against window. Same formula, steadier signal.

Where the cut goes

Walk the gaps in order. If sim(ei,ei+1)≥τ\mathrm{sim}(e_i, e_{i+1}) \ge \tau, sentence i+1i+1 joins the open chunk. If the score falls below τ\tau, close the chunk and start a fresh one at i+1i+1. Some setups rank all the gaps and cut the deepest few percent instead of fixing τ\tau; the effect is identical, a line between "same topic" and "new topic", drawn from your own score distribution rather than a borrowed magic number.

And always set a cap, Lmax⁡L_{\max }, the largest chunk you will tolerate in tokens. If a run of high scores stretches past it, force a cut at that run's lowest-similarity gap. Without the cap one slow narrative can fuse into a single giant chunk that no longer fits the embedding window.

The trade is honest. You pay for an extra embedding pass over every sentence at ingestion, which costs time and money on a large corpus, and you get chunks that read as whole thoughts, which retrieve far more cleanly. Skip this method when the document fits in one chunk anyway, or when the text has no topical structure at all, like server logs or raw transcripts. There fixed-size cuts with generous overlap win, because there is no meaning boundary to find. The method assumes cosine similarity tracks topic shifts. That assumption fails on negation, sarcasm, bare headers, and abrupt style changes, all of which move the vector without moving the topic.

Worked example

Four sentences, two about banking and two about a rover, shrunk to two dimensions so you can check every step. Finance sentences point mostly along the first axis, space sentences along the second:

e1=[4,1],e2=[4,2],e3=[1,4],e4=[2,5]e_1 = [4, 1], \quad e_2 = [4, 2], \quad e_3 = [1, 4], \quad e_4 = [2, 5]

First gap. The dot product is 4×4+1×2=184 \times 4 + 1 \times 2 = 18. The lengths are ∥e1∥=16+1=17≈4.12\lVert e_1 \rVert = \sqrt{16 + 1} = \sqrt{17} \approx 4.12 and ∥e2∥=16+4=20≈4.47\lVert e_2 \rVert = \sqrt{16 + 4} = \sqrt{20} \approx 4.47. So sim=18/(4.12×4.47)=18/18.44≈0.98\mathrm{sim} = 18 / (4.12 \times 4.47) = 18 / 18.44 \approx 0.98. Same conversation, no cut.

Second gap. Dot product 4×1+2×4=124 \times 1 + 2 \times 4 = 12. Lengths 20≈4.47\sqrt{20} \approx 4.47 and 17≈4.12\sqrt{17} \approx 4.12, product again 18.4418.44. So sim=12/18.44≈0.65\mathrm{sim} = 12 / 18.44 \approx 0.65. That is the cliff: banking just turned into space.

Third gap. Dot product 1×2+4×5=221 \times 2 + 4 \times 5 = 22. Lengths 17≈4.12\sqrt{17} \approx 4.12 and ∥e4∥=4+25=29≈5.39\lVert e_4 \rVert = \sqrt{4 + 25} = \sqrt{29} \approx 5.39, product about 22.2022.20. So sim=22/22.20≈0.99\mathrm{sim} = 22 / 22.20 \approx 0.99. Same new topic, no cut.

With the threshold τ=0.80\tau = 0.80, the scores read 0.980.98, 0.650.65, 0.990.99, and only the middle gap falls below the line. One boundary, after sentence two: Chunk A holds sentences 1 and 2, Chunk B holds 3 and 4. Those are the numbers drawn in the diagram above.

Code

The snippet below replays the worked example with the standard library only, so every printed number is one you can trace to the arithmetic above.

import math

def cosine(a: list[float], b: list[float]) -> float:    dot = sum(x * y for x, y in zip(a, b))    na = math.sqrt(sum(x * x for x in a))    nb = math.sqrt(sum(y * y for y in b))    return dot / (na * nb)

vecs = [[4.0, 1.0], [4.0, 2.0], [1.0, 4.0], [2.0, 5.0]]sims = [cosine(vecs[i], vecs[i + 1]) for i in range(3)]print([f"{c:.2f}" for c in sims])  # -> ['0.98', '0.65', '0.99']
tau = 0.80cuts = [i + 1 for i, c in enumerate(sims) if c < tau]print(cuts)  # -> [2]
bounds = [0] + cuts + [len(vecs)]names = ["S1", "S2", "S3", "S4"]groups = [names[bounds[k]:bounds[k + 1]] for k in range(len(bounds) - 1)]print(groups)  # -> [['S1', 'S2'], ['S3', 'S4']]

The only cut lands after sentence 2, exactly where the hand computation put it. In a real setup you would add the Lmax⁡L_{\max} cap here: after grouping, any group over the cap gets re-split at its own lowest-similarity gap.

Watch Out For

A threshold that splits everything or nothing

You borrow τ=0.5\tau = 0.5 from a tutorial, run it on your docs, and get confetti: every sentence its own chunk. Or you nudge it up a little and the whole manual fuses into one chunk. Either way retrieval quietly gets worse and nothing errors.

The cause is that similarity distributions move with the embedding model, the domain, and the window size. A threshold tuned on news prose means nothing on API docs. Symptom check: count chunks per document. Near one chunk per sentence means τ\tau sits too high; near one chunk per document means it sits too low.

Fix it from your own data. Score a few hundred gaps, look at the spread, and place τ\tau where the clear dips live, then confirm against retrieval quality, not vibes. RAG evaluation is where that check happens. And keep the Lmax⁡L_{\max} cap regardless, so a bad threshold degrades into uneven chunks instead of one monster.

The slow drift that never triggers a cut

Some documents never jump topics; they melt. A 20-page narrative where each sentence differs slightly from the last keeps every gap at 0.900.90 and above, so no boundary ever fires and the entire document becomes one chunk.

The symptom shows up at query time: that chunk's embedding averages twenty pages into mush, it matches vaguely or not at all, and it may exceed the embedding model's input limit, silently truncating the tail. If your longest chunks keep growing with document length, you have this bug.

Fix it with the cap from the previous section. Set Lmax⁡L_{\max} near the size your other chunks use, and when a run overflows, cut it at its lowest-similarity gap, the least-bad boundary available. If most of your corpus needs the cap, the corpus has no topic structure, and you should switch to fixed-size chunking with overlap. That switch is covered in chunking strategies, and the small-searches-big-context alternative is parent-child chunking.

The Quick Version

  • Semantic chunking draws chunk boundaries where the similarity between neighbouring sentences dips below a threshold, so each chunk holds one topic.
  • The standard score is cosine similarity between sentence embeddings: near 1 means same conversation, a sharp drop means the topic turned.
  • In the worked example, gaps of 0.98, 0.65 and 0.99 against a 0.80 threshold cut the text exactly once, between sentences 2 and 3.
  • It costs an extra embedding pass at ingestion and needs its threshold tuned on your own documents, plus a maximum-size cap so slow drifts cannot fuse a whole file.
  • Skip it for tiny documents that fit one chunk and for logs or transcripts with no topic structure; fixed-size cuts with overlap win there.