RAG Chunking Simulator: Split Documents So Retrieval Finds the Right Passage

Retrieval-augmented generation, usually called RAG, answers questions by first finding relevant passages from a document collection and then giving those passages to a language model alongside the question. The quality of the whole system depends heavily on a step that looks like plumbing: chunking, the splitting of documents into pieces for storage and retrieval. If chunks are too large, retrieval returns long passages full of irrelevant material and the model struggles to find the answer inside them. If chunks are too small, the answer is split across pieces, context is lost, and retrieval returns fragments that do not make sense alone. A RAG Chunking Simulator lets you experiment with chunk sizes, overlaps, and splitting rules on sample text before committing to a design, and that experimentation is far cheaper than rebuilding a production index.

Three decisions define a chunking strategy. The first is chunk size, usually measured in characters or tokens: how much text goes in each piece. Common starting points range from a few hundred to around a thousand tokens, but the right size depends on the documents and the questions. Dense technical documents with self-contained paragraphs tolerate smaller chunks, while narrative documents where meaning spans paragraphs need larger ones. The second decision is overlap: how much text neighbouring chunks share. Overlap of 10 to 20 percent is a common starting point, and its purpose is to protect answers that straddle a boundary, so that at least one chunk contains the full answer even when it sits near a split. The third decision is the splitting rule: whether to split at fixed character counts, at paragraph boundaries, at sentence boundaries, or by document structure such as headings. Structure-aware splitting preserves meaning better than blind fixed-size splitting, which can cut a sentence or a table in half, but it is harder to implement and behaves inconsistently across messy documents.

The trade-offs are real and measurable. Larger chunks preserve context but dilute retrieval precision, because a chunk that contains the answer plus nine irrelevant paragraphs scores similarly to a chunk that is mostly relevant, and the model must sift through more noise. Smaller chunks retrieve precisely but lose context, so a chunk containing "it rose by 20 percent" is useless without the chunk that says what "it" is. Overlap mitigates boundary problems but increases storage and the number of near-duplicate results, which can crowd out diverse passages. There is no universally optimal setting; there is only the setting that performs best on your documents and your questions, which is why simulation and evaluation matter more than received wisdom about magic numbers.

A worked example

Take a sample technical document of about 5,000 words describing a water pump: specifications, installation steps, maintenance schedule, and troubleshooting. With a fixed chunk size of 200 tokens and no overlap, the document splits into roughly 35 chunks, and a question about the maintenance interval might retrieve a chunk containing the interval but not the pump model it applies to, because the model name sat in an earlier chunk. With a chunk size of 800 tokens and 15 percent overlap, the same document splits into about 10 chunks, and the maintenance chunk now contains the model name, the interval, and the procedure together, making it far more useful to the answering model. But a question about a single specification buried in a dense table might now retrieve a long chunk where the specification is a small fraction of the content. A structure-aware approach, splitting at headings and keeping tables intact, might produce chunks of varying sizes that follow the document's own organisation, performing well on both question types. The simulator lets you try all three strategies on the same text and inspect the actual chunks, which reveals problems that abstract discussion hides, such as a table split across two chunks or a heading orphaned from its content.

Evaluation turns simulation into a decision. Build a small set of test questions with known answers from your documents, run retrieval under each candidate strategy, and score whether the retrieved chunks contain the answer and whether they contain it with enough context to use. Measure both retrieval success and the amount of irrelevant content retrieved, since a strategy that always retrieves the answer inside a huge chunk is less useful than one that retrieves it compactly. Repeat the evaluation as documents change, because a strategy tuned for manuals may fail on emails, contracts, or transcripts, which have different structures and densities. Keep the test set fixed across experiments so that improvements are real and not artefacts of changing the test.

Common mistakes

The most common mistake is choosing a chunk size from a blog post rather than from testing on your own documents. The second mistake is using zero overlap and then wondering why boundary answers fail. The third mistake is splitting blindly at fixed sizes through sentences, tables, and lists, destroying the units of meaning retrieval needs. The fourth mistake is optimising for retrieval metrics while ignoring what the answering model does with the retrieved text, since the two stages interact. The fifth mistake is chunking once and never revisiting the strategy as the document collection grows and changes. The sixth mistake is including boilerplate, headers, footers, and navigation text in chunks, which pollutes retrieval with repeated irrelevant content. The seventh mistake is assuming chunking alone fixes a bad retrieval system, when embedding quality, query formulation, and ranking matter just as much.

Honest limitations

A simulator shows how text splits; it cannot guarantee retrieval quality, because real retrieval also depends on embeddings, queries, and ranking, none of which a chunking view captures fully. Simulation also cannot judge whether retrieved passages are factually sufficient for a correct answer, since that requires the answering stage. Treat simulation as a way to eliminate bad strategies cheaply and to understand your documents' structure, then validate the surviving strategies with end-to-end tests on real questions before building on them.

Closing

Chunking is a balance: large enough to preserve meaning, small enough to retrieve precisely, with overlap to protect boundaries and splitting that respects document structure. Simulate the options, evaluate on your own questions, and revisit the choice as your collection evolves. For neighbouring jobs, the AI Token Counter measures chunk sizes in tokens and the Context Window Planner helps fit retrieved passages into a model's limits, while Context Window Planner Guide explains how retrieved context competes for space with instructions and conversation history.