Chunking Strategies

Why Does This Exist?

An embedding turns a passage into one vector, a list of numbers that captures what the passage means. That works well when the passage is about one thing. Embed a whole 50-page manual as a single vector and you get an average over every topic it touches, too blurry to match any specific query. So real retrieval systems split each document into smaller pieces called chunks first, then embed every chunk on its own. (The glossary version of the term is chunking.)

Where the cuts fall decides whether a retrieved passage reads as a complete thought or a fragment. A boundary that lands mid-sentence leaves each half without the context the other half carried, and the model receives both halves separately, in different retrieved results, or only one of them. This is the most avoidable source of bad retrieval in a RAG system, and it is set entirely by the chunking strategy.

This page covers the strategy choice itself: fixed-size cuts, structure-aware cuts, how overlap rescues split sentences, and the size tradeoff underneath everything. Two neighbours go deep on one strategy each: semantic chunking splits where the topic shifts, and parent-child chunking searches small chunks but hands the model the large one. RAG evaluation is where you measure whether a chunking change actually helped.

Think of It Like This

How It Actually Works

Call the document length $L$, measured in tokens. ($L$ is just the count of token pieces the document splits into; a token is roughly three quarters of an English word.) A strategy picks a chunk size $C$ and an overlap $O$, the number of trailing tokens each chunk repeats as the start of the next one. The window moves forward by the stride $S = C - O$ each time, and the chunk count is $N = 1 + \lceil (L - C) / S \rceil$, where $\lceil \cdot \rceil$ rounds up. If the document is shorter than one chunk ($L \le C$), there is a single chunk and none of this matters.

Fixed-size splitting and its arithmetic

Fixed-size chunking walks the window forward by $S$ tokens and cuts wherever it lands, with no regard for sentences or sections. It is cheap, predictable, and blind. Every boundary is a gamble: most land between ideas and do no harm, but some land inside a sentence and split one thought across two independently embedded chunks.

Overlap is the direct fix for that gamble. The last $O$ tokens of chunk $i$ are also the first $O$ tokens of chunk $i + 1$. So any sentence of at most $O$ tokens that straddles a boundary starts no earlier than $O$ tokens before the cut, which puts its whole span inside the next chunk. Overlap does not make cuts smart. It gives every cut a second chance.

Worked example: 1,000 tokens into 7 chunks

Take $L = 1000$ tokens with $C = 200$ and $O = 50$, so $S = 150$. Then $N = 1 + \lceil 800 / 150 \rceil = 1 + 6 = 7$ chunks. Their token spans are 1 to 200, 151 to 350, 301 to 500, 451 to 650, 601 to 800, 751 to 950, and 901 to 1000. Notice the last chunk holds only 100 tokens: unless the length divides evenly, the tail is always a runt.

Now put a 30-token definition at tokens 186 to 215. The cut at token 200 splits it: chunk 1 holds tokens 186 to 200, which is not the definition. But the sentence starts at 186, which is later than $200 - 50 + 1 = 151$, so tokens 186 to 215 sit entirely inside chunk 2 (151 to 350). The overlap rescued exactly the case it exists for. Had the definition been 80 tokens long, longer than the 50-token overlap, no overlap could have saved it, and that failure would have been structural, not bad luck.

Structure-aware splitting

Structure-aware chunking cuts where the document already changes: section headers first, then paragraphs, then sentences, then, only for a piece still too large, fixed-size splitting as the last resort. Trying the largest boundary first and falling back to smaller ones is called recursive splitting, and it is the standard practical form of this idea. Chunk sizes come out uneven, because sections are uneven, but each chunk tends to be one coherent unit that reads sensibly on its own.

Two flavours deserve names. Splitting on the document's own markup, headers and lists is rule-based and nearly free. Splitting where the meaning shifts, measured with embeddings, is semantic chunking: more accurate on messy prose, more expensive at ingestion. Either way, structure-aware splitting assumes the document has structure worth following. Chat logs, transcripts, and concatenated reviews often do not, and there fixed-size chunking with generous overlap is the honest choice, not a lazy one.

The size tradeoff that never goes away

Small chunks match precisely. A short, focused chunk produces a sharp embedding, and a specific query lands on it cleanly. But each chunk carries little surrounding context, so the model may receive the answer's sentence without the caveat two sentences away.

Large chunks keep context. The model sees the caveat, the setup, and the worked detail together. But the embedding averages more ideas, so a precise query struggles to tell the right large chunk from a wrong one that mentions the topic in passing.

There is no correct size, only a size tuned against your documents and your queries. Start near a few hundred tokens with 10 to 25 percent overlap, then move the numbers while watching retrieval scores, not while rereading samples. And when one size cannot serve both matching and context, stop tuning one size: parent-child chunking searches small chunks and hands the model the large parent instead.

One assumption underlies all of this: each chunk is judged on its own. Anything whose meaning lives far from its words breaks that assumption. Pronouns pointing three paragraphs back, "as shown above", a table split from its caption. When your documents are full of cross-references, fix the references (resolve them, prepend the section header to each chunk) before you tune anything else.

Code

Two splitters on the same text: fixed-size with overlap, and a paragraph-aware splitter. Watch where the fixed window lands.

1def fixed_size_chunks(text: str, size: int, overlap: int) -> list[str]:2    chunks: list[str] = []3    step: int = size - overlap  # window moves less than its width4    start: int = 05    while start < len(text):6        chunks.append(text[start:start + size])7        start += step8    return chunks9
10
11def paragraph_chunks(text: str) -> list[str]:12    return [p.strip() for p in text.split("\n\n") if p.strip()]13
14
15text: str = "AAA BBB.\n\nCCC DDD."16print(fixed_size_chunks(text, size=12, overlap=6))17# -> ['AAA BBB.\n\nCC', 'B.\n\nCCC DDD.', 'C DDD.']18print(paragraph_chunks(text))19# -> ['AAA BBB.', 'CCC DDD.']

The 18-character text is sliced at stride 6, so chunks start at 0, 6, and 12. The first chunk cuts "CCC" after "CC", and the third chunk is a runt fragment, "C DDD.". The overlap earns its keep in the middle chunk, which holds the whole second paragraph, "CCC DDD.", plus the tail of the first. The paragraph splitter keeps both ideas intact, because the blank line told it exactly where the boundary was.

Watch Out For

The Quick Version

• Chunking splits each document before embedding, because one vector for a whole document averages every topic it touches into blur.
• Fixed-size chunking cuts every $C$ tokens with stride $C - O$. It is cheap, predictable, and occasionally cuts a sentence in half.
• Overlap repeats the last $O$ tokens as the next chunk's start, so any sentence of at most $O$ tokens that straddles a cut still lands whole inside the next chunk.
• Structure-aware chunking cuts at section, paragraph, and sentence boundaries first and falls back to fixed-size only for oversize pieces, trading even sizes for coherent chunks.
• Small chunks match queries precisely but starve the model of context. Large chunks feed context but match vaguely. No size is correct in general.
• When one size cannot do both jobs, search small chunks and hand the model the large parent, instead of splitting the difference badly.

What to Read Next

• Semantic Chunking
• Parent-Child Chunking
• RAG Architecture
• RAG Evaluation