Chunking decides what your RAG system can find

Retrieval can only return pieces that exist. The moment you split a document into chunks, you have decided which answers are findable and which are cut in half and lost.

The step nobody reviews

Chunking is the quiet first move of every RAG pipeline. You take a document, split it into passages of some size, embed each one, and store them. It feels mechanical, so most teams pick a chunk size from a tutorial, run the whole corpus through it once, and never look again. That one unreviewed decision sets the ceiling on everything retrieval can ever do.

Think about what a chunk is. It is the smallest unit your system can hand the model. If the passage that answers a question sits entirely inside one chunk, retrieval has a shot at finding it. If that answer is spread across the boundary between two chunks, neither chunk is a good match for the question on its own, and the whole thing never comes back together.

An answer split down the middle

A support team I looked at had this exact failure. A customer asks how long the return window is for a specific product category. The answer lived in a benefits handbook, in a section where the rule and its one important exception were two paragraphs apart. Their chunker split on a fixed character count. The split landed right between the two paragraphs.

So the chunk that held the general rule looked like a strong match for the question and got retrieved. The chunk that held the exception, the part that actually changed the answer for this category, sat below the fold in the rankings and never made it into the context. The model answered with the general rule, confidently, and was wrong for exactly the cases the exception covered. Nobody had touched the prompt or the model. The passage had simply been cut in the wrong place.

That is the shape of a chunking bug. The information is in your corpus. It is just no longer in any single retrievable piece.

Too small loses context, too big drowns the signal

There is a real tension here, and it does not have one right setting. Split too small and each chunk loses the context that made it meaningful. A sentence that says “this does not apply in that case” is useless once separated from the case it refers to. Split too large and you get the opposite problem. A giant chunk covers many topics, so its embedding is a blurry average of all of them, and it matches everything weakly and nothing strongly. The one relevant sentence inside a two-page chunk gets drowned out by everything around it.

Good chunking usually means splitting on the document’s own structure instead of a blind character count. Break on headings, on sections, on logical units, and keep a rule and its exception together because a human would read them together. Some overlap between neighboring chunks helps too, so a fact that sits near a boundary appears whole in at least one piece. None of this is exotic. It just requires looking at your actual documents rather than trusting a default.

The cost of getting it once and never revisiting

Because chunking runs once at ingestion and then disappears, its failures are invisible in the usual places. The logs look fine. The model looks fine. Retrieval returns results with high confidence scores. The scores are high because the retrieved chunk really is the closest match to the query. It is just the closest match to a question whose true answer got sliced away. High confidence in the wrong piece is exactly what a chunking problem produces, and it will not show up until someone reads a wrong answer and asks where the missing detail went.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two week, $2,500 engagement, we look at how your documents are split before we look at how they are ranked. We pull real questions, trace which chunk should have carried the answer, and check whether your splitting strategy kept that answer intact or cut it apart. When it cut it apart, we fix the chunking, because no ranking model can rejoin what ingestion tore in two.

Retrieval never finds what chunking already threw away.