Chunking Strategies for RAG: Size, Overlap, and Structure
Retrieval-augmented generation (RAG) lives or dies by what you feed the retriever. Before embeddings, before reranking, before prompt engineering, there is one decision that quietly shapes everything downstream: how you split your documents into chunks. Get it wrong and your vector search returns fragments that are too vague to be useful or too bloated to be precise. Get it right and even a modest model produces grounded, accurate answers.
This guide compares the three chunking approaches you will actually use in production—fixed, recursive, and semantic—and gives you concrete rules for choosing chunk size and overlap.
Why chunking matters more than you think
Embedding models compress a passage into a single vector. If a chunk covers three unrelated topics, its vector is a muddy average that matches nothing well. If a chunk is a single sentence torn from its context, it retrieves cleanly but lacks the surrounding information the model needs to answer.
Chunking is the art of packaging one coherent idea per unit at a size your embedding model and LLM can both handle. Every strategy below is a different trade-off between simplicity, semantic coherence, and structural awareness.
Fixed-size chunking
Fixed-size chunking splits text into segments of a set token or character count, regardless of content. Split every 500 tokens, add some overlap, done.
When to use it
Fixed chunking is fast, predictable, and cheap. It works well for homogeneous text—transcripts, chat logs, plain prose without clear structure. If you are prototyping and want a baseline in an afternoon, start here.
The weakness
It cuts blindly. A fixed splitter will happily slice a sentence, a code block, or a table row in half. That mid-sentence break degrades embedding quality and produces jarring retrieved snippets. Overlap mitigates this but does not solve it.
Recursive chunking
Recursive character splitting is the pragmatic default for most teams in 2026, and it is what libraries like LangChain and LlamaIndex ship as their recommended splitter. Instead of cutting at a raw character count, it tries a prioritized list of separators—paragraphs first, then sentences, then words—and only falls back to a harder break when a segment still exceeds the target size.
Why it works
Recursive splitting respects natural boundaries whenever it can. A paragraph that fits under your size limit stays intact. A long paragraph gets split at sentence boundaries rather than mid-word. You get most of the structural benefits of semantic chunking at a fraction of the cost, because it is pure string processing with no extra model calls.
Tune the separators
The default separator list assumes prose. For Markdown, add heading markers so sections stay grouped. For code, split on function and class boundaries. For HTML, split on structural tags. Matching separators to your content type is the single highest-leverage tweak most teams skip.
Semantic chunking
Semantic chunking uses embeddings to decide where to cut. It embeds each sentence, then measures similarity between consecutive sentences and starts a new chunk when the meaning shifts—when the similarity drops below a threshold or spikes in distance.
The upside
Boundaries land exactly where topics change, so each chunk is a genuinely coherent unit. For dense, topic-shifting material—research papers, technical documentation, knowledge bases with mixed subjects—this produces the cleanest retrieval.
The cost
You pay for embedding every sentence at ingestion time, and the logic is more complex to tune and debug. For large corpora that reprocess frequently, that cost adds up. Reserve semantic chunking for high-value, relatively stable content where retrieval quality justifies the overhead.
Choosing chunk size
There is no universal number, but there is a reliable process. Anchor your decision to three constraints:
1. Your embedding model's context window
Modern embedding models handle anywhere from 512 to several thousand tokens. Chunks that exceed the model's window get silently truncated—so you lose text you thought you indexed. Always keep chunks comfortably under the limit.
2. Query specificity
Match granularity to how users ask. Fact-lookup and FAQ systems benefit from smaller chunks (roughly 200–400 tokens) that isolate single facts and retrieve precisely. Analytical or summarization use cases benefit from larger chunks (800–1200 tokens) that preserve enough context for reasoning.
3. The generation budget
Every retrieved chunk consumes context window and tokens at inference. If you retrieve the top 8 chunks and each is 1,000 tokens, that is 8,000 tokens of context before the user's question. Smaller chunks let you retrieve more diverse passages within the same budget.
A sensible starting point for general documentation is 512 tokens with recursive splitting. Measure, then adjust.
Choosing overlap
Overlap repeats a slice of the previous chunk at the start of the next, so an idea straddling a boundary is not lost. A common starting range is 10–20% of chunk size—for 512-token chunks, that is roughly 50–100 tokens.
Overlap is insurance against bad cuts, not a substitute for good boundaries. Too much overlap inflates your index, increases storage and query cost, and returns near-duplicate chunks that crowd out diverse results. If you are using recursive or semantic chunking that already respects boundaries, you can keep overlap on the lower end.
Structure-aware chunking for real documents
Production documents are rarely plain prose. Preserving structure often matters more than tuning size:
- Keep headings with their content. Prepend the section heading (or full heading path) to each chunk so an isolated passage still carries its context.
- Never split tables and code blocks. Treat them as atomic units even if they exceed your target size.
- Attach metadata. Store source, section, page, and timestamp alongside each chunk. This enables filtering, citations, and hybrid search that pure vector similarity cannot provide.
A practical decision framework
Rather than agonizing over the perfect strategy, follow this order:
Start recursive
Use recursive chunking with separators tuned to your content type, 512-token chunks, and 15% overlap. This handles most cases well and is cheap to run.
Measure retrieval quality
Build a small evaluation set of real questions with known correct sources. Track whether the right chunk appears in your top results. This number, not intuition, tells you if chunking is your bottleneck.
Escalate only where it pays
If evaluation shows retrieval is failing on topic-dense content, upgrade those collections to semantic chunking. If chunks are too coarse for fact lookup, shrink them. Change one variable at a time and re-measure.
Chunking is not a one-time setup—it is a tunable parameter you revisit as your corpus and query patterns evolve. Treat it like any other part of the system: instrument it, measure it, and iterate.
Ready to build real AI skills? Join the September 2026 cohort at Class For Jobs. Explore Advanced AI — a hands-on, live program to build and ship production AI applications, live and instructor-led with career support, resume help, and job-placement assistance.
Related reading
