Chunking Strategies for RAG: Split Documents the Right Way
Retrieval-Augmented Generation (RAG) systems live and die by the quality of what you retrieve. You can pair the best embedding model with a state-of-the-art LLM, but if your documents are split poorly, the retriever surfaces incomplete or noisy context — and the model hallucinates or answers vaguely. Chunking is the single most underrated lever in RAG quality.
This guide compares the three chunking approaches you'll actually use in production — fixed-size, recursive, and semantic — and gives you concrete guidance on when to reach for each.
Why Chunking Matters More Than You Think
Embedding models have context limits, and cramming an entire document into one vector destroys retrieval precision. When you embed a 30-page PDF as a single chunk, the resulting vector is an averaged blur that matches everything vaguely and nothing well.
Chunking breaks documents into retrievable units. The goal is to make each chunk semantically self-contained — big enough to carry meaning, small enough to stay focused. Get this wrong and you'll see two failure modes: chunks so small they lack context, or so large they dilute relevance and waste your context window.
Fixed-Size Chunking
The simplest approach: split text into chunks of a fixed number of tokens or characters, often with overlap.
How it works
You pick a chunk size (say 512 tokens) and an overlap (say 50 tokens). The splitter walks through the text, cutting at the boundary regardless of sentence or paragraph structure. Overlap ensures a sentence split across two chunks still appears intact in at least one.
Pros and cons
Pros: Fast, predictable, trivially easy to implement, and cheap. Chunk sizes are uniform, which keeps token budgeting simple.
Cons: It ignores meaning entirely. A fixed cut can slice a sentence, table, or code block in half, orphaning context. The overlap helps but wastes storage and can introduce duplicate retrievals.
When to use it
Fixed-size chunking is a solid baseline and often good enough for uniform, prose-heavy content like transcripts or articles. Start here, measure retrieval quality, then upgrade only if the numbers demand it.
Recursive Chunking
Recursive character splitting is the pragmatic default in most RAG frameworks in 2026, and for good reason. It respects document structure without the cost of semantic analysis.
How it works
You define a hierarchy of separators — typically paragraphs (\n\n), then lines (\n), then sentences, then words. The splitter tries to break on the largest separator first. If a resulting chunk is still too big, it recurses to the next separator down until every chunk fits your target size.
The result: chunks that respect natural boundaries whenever possible, falling back to smaller units only when necessary. A well-structured document keeps its paragraphs intact; a dense wall of text still gets split cleanly.
Pros and cons
Pros: Balances structure-awareness with speed and cost. It rarely cuts mid-sentence, produces coherent chunks, and requires no extra model calls.
Cons: It's still fundamentally length-based. Two adjacent paragraphs about completely different topics might end up in the same chunk if they fit the size budget, and one long topic might be arbitrarily split.
When to use it
Reach for recursive chunking as your general-purpose choice for markdown, HTML, documentation, and mixed content. For code and structured formats, use language-aware separators (function definitions, class boundaries) so you don't split logical units.
Semantic Chunking
Semantic chunking splits based on meaning rather than length. Instead of asking "how many tokens," it asks "where does the topic change?"
How it works
The text is first divided into sentences. Each sentence (or small group) is embedded. The splitter then compares embeddings of adjacent sentences using cosine similarity. When similarity drops below a threshold — signaling a topic shift — it creates a new chunk boundary. The result is variable-length chunks that each cover a single coherent idea.
Pros and cons
Pros: Produces the most topically coherent chunks, which typically improves retrieval precision, especially for documents that jump between subjects. Each chunk maps cleanly to a concept.
Cons: It's the most expensive and slowest option because you embed every sentence before you even index. Threshold tuning is finicky — too sensitive and you fragment, too loose and you merge unrelated ideas. For large corpora, the preprocessing cost adds up.
When to use it
Use semantic chunking when retrieval quality is critical and documents are heterogeneous — research papers, legal contracts, knowledge bases spanning many topics. If your recursive baseline is missing relevant context or retrieving off-topic chunks, semantic splitting is the natural upgrade.
Choosing the Right Strategy
There's no universally best method — the right choice depends on your content, latency budget, and quality bar. A practical decision path:
Start with recursive chunking at around 512 tokens with 10–15% overlap. It's the best effort-to-quality ratio for most applications.
Drop to fixed-size if your content is uniform prose and you want maximum indexing speed and simplicity.
Upgrade to semantic only when evaluation shows recursive chunking is hurting retrieval on topically diverse documents.
Don't skip evaluation
Never pick a chunking strategy by intuition alone. Build a small test set of representative questions with known correct source passages, then measure retrieval metrics like context recall (did the right chunk get retrieved?) and context precision (how much of what was retrieved is relevant?). Run each strategy against the same set and let the numbers decide.
Practical Tips That Move the Needle
Tune chunk size to your embedding model and query style. Short factual questions favor smaller chunks; broad analytical queries benefit from larger ones. Test 256, 512, and 1024 tokens before committing.
Add metadata to every chunk. Store the source document, section heading, page number, and position. This enables filtering, citation, and better reranking downstream.
Preserve structure in preprocessing. Convert PDFs and HTML into clean markdown first so headings and lists survive. Handle tables and code blocks as atomic units rather than letting the splitter carve them up.
Consider hierarchical or parent-document retrieval. Retrieve on small, precise chunks but feed the LLM the larger parent section they belong to — you get precision in matching and completeness in context.
Layer in a reranker. Even great chunks benefit from a cross-encoder reranking pass that reorders retrieved candidates before they reach the model.
Chunking isn't a one-time decision you make and forget. Treat it as a tunable parameter of your RAG pipeline: start simple, measure honestly, and escalate only when your metrics tell you to.
Ready to build real AI skills? Join the September 2026 cohort at Class For Jobs. Explore Advanced AI — a hands-on, live program to build and ship production AI applications, live and instructor-led with career support, resume help, and job-placement assistance.
Related reading
