How to Choose the Right Chunking Strategy for RAG

Why chunking matters

A retrieval system cannot return information it failed to index in a useful unit. If an answer spans two chunks, a query may retrieve only one half. If a chunk combines many unrelated topics, its embedding can become a weak average of several meanings. The result is a retrieval problem that often gets mistaken for an LLM problem.

The right objective is not a universal token count. The objective is to maximize the probability that the evidence needed for an answer is retrieved together while keeping each chunk specific enough to rank well.

• Treat the chunk as a retrieval boundary, not just a text-size limit.
• Preserve document structure and metadata alongside the text.
• Evaluate retrieval separately from generation.

Four practical strategies

Fixed-size chunking is simple and useful as a baseline, but it can cut sentences, tables, or code blocks. Recursive splitting improves the baseline by trying preferred separators before falling back to smaller boundaries. Semantic chunking groups text based on meaning, but it adds computation and can be harder to reason about. Structure-aware chunking uses headings, sections, paragraphs, tables, and other document boundaries and is often the most natural starting point for technical and business documents.

For manuals, policies, reports, and documentation, structure-aware splitting usually preserves meaning better than blindly counting characters. For irregular sources, combine structural rules with a size limit so one huge section cannot create an unmanageable chunk.

• Fixed size: fast baseline.
• Recursive: practical general-purpose option.
• Semantic: useful when topic boundaries are important.
• Structure aware: strong choice for structured documents.

Keep context outside the chunk

A chunk should not have to carry every global fact as prose. Store document title, section path, source URL, page number, version, effective date, tenant, product, and other retrieval constraints as metadata. When a document says that all values are in thousands or that a policy applies only to a region, preserve those facts through metadata or contextual headers so the generator is not forced to infer them.

Parent-child retrieval is another useful pattern: retrieve small child chunks for precision, then provide the larger parent section to the model. This keeps retrieval focused without starving generation of surrounding context.

Evaluate chunking instead of guessing

Create a small labeled set of realistic questions and identify the evidence that should be retrieved. Compare chunking strategies using recall at k, precision of retrieved evidence, and downstream answer quality. Keep generation settings fixed while testing retrieval so you can attribute changes correctly.

Also inspect failure cases manually. A metric can tell you that a strategy improved; examples tell you why. Look for answers split across boundaries, tables separated from captions, references that lost their antecedent, and duplicate chunks that crowd out useful evidence.

• Start with 30–100 representative questions if you are prototyping.
• Track retrieval recall at the k values your application actually uses.
• Keep an error log with the document, query, retrieved chunks, and expected evidence.

Production checklist

Before shipping, version your chunking configuration, embedding model, and index. If you change chunk boundaries or embeddings, treat it as an index migration and run regression evaluations before switching traffic. Monitor no-hit queries, retrieval scores, citation failures, and index freshness. Chunking is infrastructure, so changes should be tested like code.

• Preserve source locations for citations.
• Make deletes and document updates propagate to the index.
• Keep a regression set for every retrieval change.