Skip to content
AI360XpertBeta

Parent-Child Chunking

Search tiny child chunks for precise matches, then hand the model the full parent passage they came from. Sharp matching, full context.

A query matches one 100-token child chunk, and the system follows its parent pointer to send the full 400-token parent passage to the model
A query matches one 100-token child chunk, and the system follows its parent pointer to send the full 400-token parent passage to the model

Why Does This Exist?

Every RAG pipeline hits the same sizing fight. Cut documents into small pieces and each embedding stays sharp: a 100-token chunk about heart chambers matches a question about heart chambers cleanly. But hand that lone fragment to the model and it answers without the caveats sitting two sentences away. Cut into big pieces and the model gets the full picture, yet each embedding averages many ideas together, so a precise question can no longer pick the right passage out of the pile.

One size cannot serve both jobs. Matching wants small. Reading wants big.

Parent-child chunking stops compromising on a single size. You index small child chunks for the search step and store large parent chunks for the reading step, with each child carrying a pointer to the parent it came from. The query matches a child. The model receives the parent. This page covers that two-level trick end to end. The single-size choice it replaces is chunking strategies, the boundary question underneath it is semantic chunking, and the short glossary form is chunking.

Think of It Like This

A textbook index that hands you the chapter

You own a thick textbook and someone asks when a famous landing happened. You do not reread the book. You flip to the index at the back, find one sharp line ("Normandy landing: page 412, paragraph 2"), and jump straight there. The index is built for finding: tiny entries, exact pointers.

But you do not read only that one sentence. Once the pointer lands, you read the whole section around it: the plan, the weather, what happened next. The sentence told you where to look. The section tells you what it meant.

Child chunks are the index entries. Parent chunks are the sections. The pointer between them is the page number.

The analogy stops in one place. An index is written by hand and matched by spelling. Child chunks are matched by meaning: the query and each child are both turned into embeddings, and the closest vectors win. Nothing here looks up words. It compares positions in vector space.

How It Actually Works

Two stores, built at index time, used in sequence at query time.

At ingestion you split twice. First split each document into parents of size PP tokens and keep them in a plain document store, keyed by id: parent_A, parent_B, and so on. No vectors needed there. Then split every parent into children of size CC tokens, embed each child, and store the vectors in the vector database with one extra metadata field: the id of the parent the child came from. That pointer is the whole mechanism. Everything else is plumbing.

At query time the steps run in order: embed the query, rank children by similarity, read the winner's parent pointer, fetch that parent from the document store, and place the parent (not the child) into the prompt. Search small, read big.

The two-level split

Call the document length LL tokens (LL is just the count of token pieces; a token is roughly three quarters of an English word). The parent count is N=L/PN = L / P, and each parent holds P/CP / C children. Pick PP for reading: large enough to hold one complete idea with its caveats, small enough to fit comfortably in the prompt. Pick CC for matching: small enough that one embedding means one thing. A common working shape is P=400P = 400 with C=100C = 100, four children per parent.

Worked example: one hit returns 400 tokens

Take a document of L=800L = 800 tokens. With P=400P = 400 you get N=800/400=2N = 800 / 400 = 2 parents, A and B. With C=100C = 100 each parent splits into 400/100=4400 / 100 = 4 children, 8 children in total. Every child embedding covers one focused span, so the query "how many heart chambers" lands squarely on child 2 of parent A instead of smearing across a whole section.

The system reads child 2's pointer (parent_A), fetches parent A from the document store, and sends 400 tokens to the prompt instead of 100. That is 4 times the context for the same match. Parent B is never touched: its children scored lower, so it stays on the shelf. The diagram above draws exactly this: four 100-token children under a 400-token parent A, the hit on child 2, and the full parent travelling to the prompt.

Auto-merging when several children agree

Sometimes the top results cluster: three of the top five children all point at parent A. That clustering is evidence the whole section matters, not just one sentence. Auto-merging acts on it with a threshold rule: if at least TT retrieved children share a parent (often T=2T = 2 or 33), return the parent once instead of returning each child. If only one child of a parent matched, keep just that child and save the tokens. The threshold is the dial between precision and context, and like every retrieval setting it belongs under measurement with RAG evaluation practice.

The tradeoff never disappears, it just moves. Parents cost prompt tokens: fetching 2 parents at 400 tokens each spends 800 tokens of context to answer from what matched in 200. That is the right trade whenever the surrounding passage carries the caveat, the dosage, or the condition the bare sentence lacks. When it does not, when answers truly live in single sentences, skip the hierarchy: plain small chunks plus reranking is cheaper and just as good. The method also assumes each parent reads sensibly on its own. If your documents are full of pointers ("as shown above", pronouns reaching three paragraphs back), resolve those references before you tune anything else, or every parent arrives with holes in it.

Code

A toy retrieval loop at word scale: four children point at one parent, overlap picks the winner, and the parent (not the child) goes to the model. The five-word children stand in for 100-token chunks; the shape is what matters.

doc_store: dict[str, str] = {    "parent_A": "The heart has four chambers. Two atria receive blood. "                "Two ventricles pump it out. Valves keep the flow one way.",}
children: list[dict[str, str]] = [    {"id": "child_1", "text": "The heart has four chambers.", "parent_id": "parent_A"},    {"id": "child_2", "text": "Two atria receive blood.", "parent_id": "parent_A"},    {"id": "child_3", "text": "Two ventricles pump it out.", "parent_id": "parent_A"},    {"id": "child_4", "text": "Valves keep the flow one way.", "parent_id": "parent_A"},]

def search_children(query: str) -> dict[str, str]:    query_words: set[str] = set(query.lower().split())    best: dict[str, str] = children[0]    best_overlap: int = -1    for child in children:        overlap: int = len(query_words & set(child["text"].lower().split()))        if overlap > best_overlap:            best = child            best_overlap = overlap    return best

def retrieve(query: str) -> str:    hit: dict[str, str] = search_children(query)    print(f"hit: {hit['id']}")  # -> hit: child_1    return doc_store[hit["parent_id"]]  # follow the pointer, return the parent

context: str = retrieve("how many chambers does the heart have")print(len(context.split()))  # -> 20print(context.startswith("The heart has four chambers"))  # -> True

The query shares two words ("the", "heart") with child 1 and fewer with the rest, so child 1 wins and its pointer resolves to parent_A. The model receives all 20 words of the parent, including the atria and valve sentences the matching child never mentioned. That surplus is the entire point: the match was precise, and the context came along anyway.

Watch Out For

Fetching five parents for a top-5 search

The default retrieval loop returns the top kk children and swaps each for its parent, so k=5k = 5 with distinct parents spends 5×400=20005 \times 400 = 2000 tokens of prompt on one query. The symptom arrives as a bill and as latency: prompts balloon, answers slow down, and long parents crowd out the second and third genuinely useful passages.

Cap parents before you fetch them. Deduplicate parent ids first, keep the top one or two parents, and budget the token spend the same way you budget everything else in the prompt. Auto-merging with a threshold of 2 or 3 earns its keep here: clustered hits collapse onto one parent instead of multiplying.

Children pointing at parents that no longer exist

The vector store and the document store are two systems that must agree, and re-chunking breaks the agreement silently. You re-split with a new size, rewrite the vectors, but the old document store keeps serving stale parents under recycled ids. The symptom is specific: retrieved passages that almost answer the question but describe the wrong section, or lookups that miss outright. Nothing errors. The context is just wrong.

Write parents and children in one atomic step and re-index them together, and store the chunking settings (sizes, overlap, version) alongside each parent so a mix of generations is detectable instead of silent. If the settings changed, rebuild both stores, never one.

The Quick Version

  • Parent-child chunking indexes small child chunks for matching and serves large parent chunks for reading, joined by a parent pointer on every child.
  • Small chunks match precisely but starve the model. Large chunks feed context but match vaguely. This pattern assigns each job to the size built for it.
  • One working shape: 400-token parents split into four 100-token children each, with the query's hit child resolving to its 400-token parent.
  • When several top hits share a parent, auto-merging returns the parent once; a lone hit stays a lone child to save tokens.
  • Deduplicate and cap parents before fetching, or a top-5 search can spend 2000 prompt tokens on a single query.
  • Rebuild the vector store and the document store together whenever chunk sizes change, or children will point at stale parents.