RAG Chunking Strategies That Actually Improve Retrieval
Chunking is the quiet bottleneck in most Retrieval-Augmented Generation systems. Teams spend weeks tuning prompts and swapping embedding models, then wonder why the retriever keeps surfacing irrelevant passages. More often than not, the culprit is how the source documents were split before they were ever embedded. Get chunking wrong and no amount of prompt engineering will save you.
This guide compares the three RAG chunking strategies you will actually reach for in production—fixed-size, recursive, and semantic—and gives concrete guidance on when each one wins.
Why Chunking Determines Retrieval Quality
An embedding model compresses a chunk of text into a single vector. That vector has to represent everything in the chunk. If a chunk contains three unrelated topics, its vector becomes an average that matches none of them well. If a chunk splits a single idea across two pieces, neither piece carries the full meaning.
Good chunks share two properties: they are semantically coherent (one idea per chunk) and appropriately sized for both the embedding model's context window and the answer you expect to generate. Chunking is where you trade off precision against recall. Smaller chunks give sharper matches but risk losing context; larger chunks preserve context but dilute relevance.
Fixed-Size Chunking
Fixed-size chunking splits text into segments of a set token or character count, usually with an overlap of 10–20% so that ideas straddling a boundary appear in both neighbors.
How it works
You pick a chunk size (say 512 tokens) and a stride. The splitter walks through the document and cuts at each boundary regardless of sentence or paragraph structure. Overlap ensures a sentence cut in half still lives intact in the adjacent chunk.
When it wins
Fixed-size chunking is the right default when your corpus is large, uniform, and you need predictable performance. Log data, transcripts, chat histories, and other flat text without strong structural markers respond well to it. It is fast, deterministic, and trivial to reason about for capacity planning, since every chunk consumes a known number of tokens.
Where it fails
It will happily cut a table in half, split a code function across two chunks, or separate a heading from the paragraph it introduces. For structured or technical documents, that fragmentation shows up as retrieval that returns partial answers.
Recursive Chunking
Recursive chunking respects document structure. It tries to split on the largest natural boundary first—say, double newlines for paragraphs—and only falls back to smaller delimiters (single newlines, sentences, then words) when a chunk still exceeds the target size.
How it works
You supply an ordered list of separators, for example ["\n\n", "\n", ". ", " "]. The splitter attempts the first separator; if the resulting pieces are still too large, it recurses into each oversized piece with the next separator down the list. The result is chunks that stay under your size ceiling while breaking at the most meaningful available point.
When it wins
Recursive chunking is the best general-purpose choice for most real-world documents: technical docs, articles, wikis, product manuals, and Markdown. It keeps paragraphs intact when it can and only fragments when a passage is genuinely too long. For code, use a language-aware variant that splits on function and class boundaries rather than blank lines.
Where it fits in your pipeline
If you are starting a new RAG project and are not sure where to invest, start with recursive chunking at around 500–800 tokens with 50–100 tokens of overlap. It gives you 80% of the benefit of more advanced approaches with almost none of the cost or complexity.
Semantic Chunking
Semantic chunking splits based on meaning rather than structure or size. It embeds individual sentences, then measures the similarity between consecutive sentences and cuts a new chunk wherever the topic shifts—that is, wherever the similarity between neighbors drops below a threshold.
How it works
The document is first broken into sentences. Each sentence (often with a small window of neighbors for context) is embedded. The algorithm walks the sequence, tracking the distance between adjacent embeddings. A large jump in distance signals a topic boundary, and a new chunk begins there. The result is variable-length chunks that each cover a single coherent idea.
When it wins
Semantic chunking pays off when topics inside a document vary in length and shift abruptly—research papers, meeting notes covering multiple agenda items, or knowledge-base articles that jump between concepts. Because each chunk maps cleanly to one topic, the embeddings are sharper and retrieval precision improves noticeably.
The trade-offs
Semantic chunking is expensive. You are running an embedding pass over every sentence just to decide where to cut, which adds latency and cost at ingestion time. The threshold also needs tuning per corpus, and a poorly chosen threshold produces either giant chunks or a flood of tiny ones. Treat it as an optimization you apply after a simpler strategy proves insufficient—not a starting point.
Choosing the Right Strategy
A practical decision path:
Start with recursive chunking. For the vast majority of document types it delivers coherent, well-sized chunks with minimal effort. It should be your default.
Use fixed-size when volume and speed dominate. If you are ingesting millions of near-uniform records where structure carries little meaning, fixed-size with overlap is cheaper and predictable.
Reach for semantic chunking to fix precision problems. If evaluation shows the retriever pulling chunks that mix relevant and irrelevant content, and your documents have variable topic lengths, semantic chunking is worth the extra ingestion cost.
Tuning Details That Matter More Than the Strategy
The strategy you pick matters, but so do the knobs around it:
Chunk size should match your embedding model and query type. Short factual questions favor smaller chunks; explanatory answers favor larger ones. Check your embedding model's optimal input length rather than guessing.
Always keep some overlap. Even 10% prevents ideas from being orphaned at boundaries. Zero overlap is a common and avoidable source of missed retrievals.
Attach metadata to every chunk. Store the source document, section heading, and position. This enables filtering and lets you reconstruct context when a chunk alone is ambiguous.
Measure, do not assume. Build a small evaluation set of real questions with known-correct source passages. Compare strategies against retrieval metrics like recall@k and context precision. The right chunking approach for your data is the one your evaluation numbers endorse, not the one that sounds most sophisticated.
Chunking is not a set-and-forget decision. As your corpus grows and question patterns emerge, revisit your splitter, re-run your evaluations, and adjust. The teams that treat chunking as a tunable, measurable part of the pipeline consistently ship RAG systems that retrieve the right context—the ones that treat it as an afterthought spend months debugging symptoms instead.
Ready to build real AI skills? Join the September 2026 cohort at Class For Jobs. Explore Advanced AI — a hands-on, live program to build and ship production AI applications, live and instructor-led with career support, resume help, and job-placement assistance.
Related reading
