Parent-child retrieval: retrieve small, serve big

Lesson 08 — the chunk that scored highest is not always the chunk the LLM needs. Retrieval precision and serving scope are two decisions, not one.

The hidden assumption in most RAG systems

The default RAG pipeline does one chunking pass. Whatever you chunked is what you embed, what you retrieve, what you rerank, and what you put in the prompt. The same slice of text plays five different roles.

It works until it doesn’t. When the chunks are big — say, full articles — retrieval gets fuzzy because the relevant sentence is buried under five sentences that aren’t. When the chunks are small — say, two-sentence paragraphs — retrieval gets sharp but the LLM gets handed a slice that doesn’t carry enough context to answer the question. A clause that says “the Authority shall publish a list” is not useful without knowing which Authority, what list, and under what regulation.

What we retrieve and what we serve can be different — and they should be, when the chunk that scored highest is too small to interpret on its own.

Two indexes, one corpus

Parent-child retrieval splits the chunking decision. Every parent (here, a full article from LN 226, LN 227, or Cap. 586) is split into children of about 50 tokens each — small enough that retrieval can pin the exact sentence, big enough that the embedding has something to work with. The child index is what BM25, dense, and the reranker search against. The parent index is what the prompt builder serves.

Each child carries its char offsets back into the parent, so once the reranker picks a child, you can fetch the surrounding article and put that in the prompt. The small chunk got the chunk’s attention; the big chunk gives the model its context.

Watch the trade-off

Pick a query. The left column shows the top five small children the cross-encoder picked from the hybrid top-20. Click any of them and the right column renders the child’s parent article — with the child slice highlighted inline. The toggle at the top of the right column switches “what would be served to the LLM” between the small chunk only and the full parent, with a token-cost on each option so the trade-off is honest.

What you’ll feel

• Sometimes parent-child is a 30× scope expansion. The cross-doc query (“data protection obligations for AI deployers”) hits a 126-token child whose parent is 3,890 tokens — that’s the entire LN 227 Article 2 definitions block. Serving the small chunk leaves the model guessing what “the Authority” refers to. Serving the parent puts the definitions in the prompt and the answer becomes trivially correct.
• Sometimes parent-child does nothing. The concept query (“how does attention work in a transformer”) hits a 435-token chunk whose parent is also 435 tokens — the chunker already produced a chunk small enough to be precise but big enough to interpret. Parent-child gains nothing on this query. That is not a bug. That is the architect knowing when the pattern doesn’t pay off.
• The interesting cases are in between. The paraphrased and acronym queries land 15–18× expansions. The small chunk found the right sentence; the parent gives the article it sits in — definitions, scope, surrounding sub-clauses. The model gets the slice that won and the context it needs to interpret the slice.

Why this is not “just use bigger chunks”

The first thing every RAG project asks when it sees a wrong answer is “should I make the chunks bigger?” The honest answer is: that fixes the serving scope and breaks the retrieval precision. A 2,000-token chunk gets the model enough context but is harder for an embedding to summarise into a single vector — your dense retrieval starts ranking by the chunk’s average topic rather than by the specific sentence that matters. Recall drops.

Parent-child lets you stop choosing. Search small for precision; serve big for scope. The cost is one extra index and a lookup step at retrieval time. Compared to a re-embedding pass or a fine-tune, that is almost free.

The architect’s decision rule

• Use parent-child when retrieval needs to point at specific sentences inside structurally large documents. Legal text, regulation, contracts, long technical specs. The unit a regulator cites is “Article 7 paragraph 2”; the unit the dense retriever can rank well is “one sentence”. They are different.
• Skip parent-child when the corpus is already shaped like the answers. Help-centre articles, knowledge-base FAQs, product descriptions. If a single chunk already answers a question end-to-end, splitting it just gives you two unrelated indexes to maintain.
• The deciding question is: would the model need the surrounding context to interpret the hit? If yes — definitions in the article above, conditions in the sub-clause below, the heading the paragraph sits under — parent-child is the cheapest way to give the model both halves.

What comes next

Lessons 05 to 08 are how to build a retrieval system that puts the right chunks into the prompt with the right context around them. Lesson 09 — deferred to Month 2 — is how to diagnose what went wrong when the model still gives the wrong answer. Retrieval failure, chunking failure, or generation failure. The architect’s diagnostic vocabulary, made into a quiz over recorded RAG traces.

Markdown version of this page: https://mokh.tech/lessons/parent-child-retrieve-small-serve-big.md