Why parent-child chunking beats flat RAG on dense documents

Most RAG tutorials chunk documents into fixed 512-token windows with overlap and call it a day. That works for blog posts. It fails on dense technical manuals, legal briefs, and API references where meaning lives in section hierarchy.

The fragmentation problem

When you split a document mid-paragraph, the embedding model encodes a fragment without its surrounding context. A child chunk about "retry policy configuration" loses the parent section heading "Network Timeouts" — and retrieval returns the chunk without the framing a downstream LLM needs to answer faithfully.

Flat chunking also creates duplicate content when overlap is high, inflating index size and diluting BM25 scores.

Parent-child indexing

Parent-child chunking treats each logical section as a parent node and splits content into smaller child chunks linked bidirectionally. At index time:

1. Parse documents into sections (headings, page breaks, structural markers).
2. Create parent records for each section with full section text.
3. Split parents into child chunks sized for embedding models.
4. Store parent ID on every child for expansion at query time.

At query time, retrieval runs against child chunks (smaller, more precise embeddings), then expands each hit to include its parent context before ranking or passing to the LLM.

Hybrid search matters here

Parent-child chunking solves context fragmentation, but it doesn't solve vocabulary mismatch. Technical documents use exact terminology — error codes, legal citations, product names — that embedding models handle inconsistently.

Combining BM25 with vector search catches both semantic paraphrases and exact matches. In practice, I tune alpha weighting per corpus using a labeled eval set rather than guessing.

Measuring the lift

Don't ship chunking strategy changes without benchmarks. I evaluate with:

• Recall@k against labeled question-chunk pairs
• Answer faithfulness — does the retrieved context actually support the generated answer?

On technical manual corpora, parent-child chunking with section-aware splits improved Recall@5 by roughly 18% over flat 512-token windows. The gain was largest on questions that referenced section-level context ("What does the installation guide say about dependencies?").

When not to use it

Parent-child adds indexing complexity. For short, unstructured text — support tickets, chat logs, tweet-sized content — flat chunking is simpler and sufficient. Use parent-child when document structure carries meaning the retrieval step needs to preserve.