I have rebuilt chunking pipelines for four client RAG systems in the past two years. In every case the LLM was fine, the embedding model was fine, and retrieval was still bad. The cause was always the same. Someone split the documents with a default setting on day one, shipped it, and never looked at the chunks again. Chunking is the highest-impact knob in a RAG system, and it is the one most teams touch exactly once.
This post compares the five chunking strategies I see in production, shows where each one breaks, and ends with the evaluation loop that tells you whether a change actually helped. If you are still planning the broader system, start with our guide on building a RAG system for a business knowledge base and come back.
Why Chunking Decides Retrieval Quality
A RAG system can only answer from the text it retrieves. Retrieval works by embedding each chunk into a vector and matching it against the embedded question. That means a chunk has two jobs that pull against each other. It must be small and focused enough that its embedding represents one idea clearly, and it must be large and complete enough that the LLM can actually answer from it. Every chunking strategy is a different bet on that trade-off.
Bad chunking produces three failure modes I see constantly. Orphan chunks: a sentence like “This fee applies after the trial period” with no subject, so the embedding matches nothing useful. Shredded context: a procedure split across three chunks, so the LLM gets step 2 and step 4 but never the whole answer. Diluted chunks: 2,000 tokens covering four topics, so the embedding is an average of everything and a match for nothing.
If you want the math behind why focus matters, our guide on building a RAG system for a business knowledge base covers how embeddings and similarity scoring work. The short version: an embedding is a lossy summary, and the more topics you cram into one vector, the worse that summary gets.
The Five Strategies and Where Each Breaks
Fixed-Size Chunking
Split every N characters or tokens, with some overlap. This is what most tutorials ship. It is fast, predictable, and easy to reason about. It is also the worst default for real documents.
Fixed-size splitting does not know what a sentence is. It will cut a paragraph in half, separate a table header from its rows, and slice a function body at line 47 of 90. You get orphan chunks by construction. Where it works: uniform, low-structure text like transcripts, chat logs, and sensor narratives, where any window is about as meaningful as any other. For anything with real structure, skip it.
Recursive Character Splitting
Try to split on the largest boundary first (double newline), and if a piece is still too big, fall back to smaller boundaries (newline, then sentence, then space). This is the default in LangChain’s text splitter docs, and it is my default for plain prose too.
It breaks in two places. First, documents with weak paragraph structure, like a legal agreement that is one 4,000-word section, fall back to sentence splits and you are back to context shredding. Second, code and tables: a recursive splitter that does not know markdown fences will happily split inside a code block because the blank lines look like paragraph breaks.
Sentence-Based Chunking
Split on sentence boundaries with a proper tokenizer, then group sentences until you hit your size target. Cleaner than recursive for prose because no chunk ever ends mid-sentence.
The failure mode is that sentence detection is harder than it looks. Legal citations (“Inc. v. Smith”), numbered lists, and technical docs full of version strings (“fixed in 2.4.1”) confuse naive splitters, which is why production libraries like spaCy exist instead of a regex. Bullet lists are worse: many have no terminal punctuation at all, so a sentence splitter sees one giant “sentence” and hands you a 1,500-token chunk anyway. Test your splitter on your actual documents before trusting it.
Semantic Chunking
Embed consecutive sentences, measure the similarity between neighbors, and cut where similarity drops, on the theory that a topic shift shows up as a similarity cliff. When it works, the chunks line up with actual ideas, which is exactly what embedding-based retrieval wants.
The costs are real. You pay for an embedding call per sentence at ingest time, which gets expensive on a large or frequently updated corpus. The breakpoint threshold is a magic number you will tune per document type. And the chunk sizes are uneven by design: one chunk might be 80 tokens, the next 900, which wrecks your token budgeting downstream. I have also seen it fail on documents that stay on one topic for pages, like a single contract clause, because the similarity never drops and you get one giant chunk. Use it when document topics genuinely shift and you can afford the ingest cost, not as a default.
Document-Structure-Aware Chunking
Split on the structure the document already has: markdown headings, HTML sections, PDF layout blocks, or function boundaries in code. A markdown splitter keeps each section intact, splits oversized sections recursively, and records the heading path as it goes.
This is the best default for knowledge bases, documentation, and wikis, because humans already organized the content into answer-sized units. It breaks on documents with fake structure: PDFs converted to markdown where every line became a heading, OCR output with no structure at all, and web pages where the DOM is divs inside divs with no semantic tags. Garbage structure in, garbage chunks out. Check a sample of conversions by hand before you trust the pipeline.
Chunk Size Versus Model Context
Context windows are huge now, and a common reaction is to make chunks huge too. That is a mistake, for two reasons.
First, retrieval quality drops as chunks grow. A 1,500-token chunk about four topics has a blurrier embedding than a 300-token chunk about one. Second, stuffing more context into the prompt costs money and accuracy. Models still attend unevenly to long prompts, a challenge we address in our context engineering for AI agents guide, and every extra chunk is tokens you pay for on every query. The token math gets ugly at scale, and it feeds directly into your overall AI ROI picture.
My working ranges, in tokens: 200 to 400 for FAQ-style question answering, 400 to 700 for documentation and policy text, and up to 1,000 for narrative content where answers span paragraphs. Start small. You can always hand the LLM more chunks; you cannot un-blur a bad embedding.
Overlap, Headings, and Metadata
Three cheap upgrades that outperform most splitter swaps.
Overlap: repeat 10 to 15 percent of each chunk at the start of the next one, on a sentence boundary, not a raw token count. Overlap rescues facts that live near a boundary. More than 20 percent inflates storage and produces near-duplicate hits that crowd your top-k. For step-by-step procedures, overlap on the higher side, because steps depend on their neighbors.
Headings: prepend the heading path to the chunk text before embedding. “Refunds Policy > Digital Goods > Exceptions: Customers may request…” retrieves far better than the bare sentence, because the embedding now encodes what the text is about, not just what it says. This one change fixed a client system where 40 percent of failed lookups were orphan chunks.
Metadata: store source title, section, URL, and date on every chunk, but keep them out of the embedded text unless they add meaning. Metadata gives you filtering (search only current policies), citations in the answer, and debugging when a bad chunk surfaces. Citations also matter downstream, because retrieved chunks with source URLs are how you catch AI hallucinations before they reach customers.
Parent-Child Retrieval
The cleanest resolution of the small-versus-large tension is to stop choosing. Embed small chunks for search, but return their larger parents to the LLM. Index child chunks of 128 to 256 tokens, each pointing at a parent of 512 to 1,024 tokens. The child gives you a sharp embedding match; the parent gives the model enough surrounding context to answer well.
Both LangChain’s ParentDocumentRetriever and LlamaIndex’s node hierarchies implement this pattern, and the Pinecone chunking guide describes the same idea under late chunking and context enrichment. The underlying retrieval primitive is still approximate nearest-neighbor search over embeddings, which our vector search fundamentals guide and the pgvector documentation cover well if you want to run it inside Postgres instead of a dedicated vector database. Two practical notes. Deduplicate parents before stuffing the prompt, because three matching children often share one parent. And cap total parent tokens per query, because three 1,000-token parents plus system prompt plus history adds up fast.
Parent-child adds moving parts: a document store alongside the vector store, and join logic in retrieval. Worth it once your corpus passes a few hundred documents or your eval shows context-starved answers. Before that, plain structure-aware chunks with heading prefixes get you most of the benefit.
Evaluate Chunking Changes with a Test Set, Not Vibes
Every chunking claim in this post, including mine, should be tested against your corpus. The loop is simple.
Build 30 to 60 real questions, each labeled with the passage that should be retrieved. Use questions from actual users, support tickets, or sales calls, not ones you invented while staring at the docs. Then measure recall: for each question, does the correct passage appear in the top-k retrieved chunks?
def hit_rate(questions, retriever, k=5):
hits = 0
for q in questions:
chunks = retriever.search(q.text, k=k)
retrieved = {c.id for c in chunks}
if retrieved & set(q.gold_chunk_ids):
hits += 1
return hits / len(questions)Run that against your current pipeline, write the number down, then change exactly one thing: the splitter, the size, the overlap, or the heading prefix. Rerun. I have watched a switch from fixed 512-token chunks to recursive 400 with heading prefixes move top-5 recall from 61 percent to 84 percent on a 90-document policy corpus. I have also watched semantic chunking move it by two points on a documentation set where structure-aware splitting already did the work. Your corpus decides, not the strategy’s reputation.
Two rules keep the loop honest. Never tune against questions you generated from the chunks themselves, because you will teach to the test. And when recall improves, spot-check that answers improved too, because retrieval is a means, not the end.
When NOT to Over-Engineer Chunking
If your corpus is under a hundred documents and you are still proving the use case, ship recursive splitting at 512 tokens with 64 overlap and spend your time on the test set instead. A prototype with an honest eval beats a clever pipeline with none. Semantic splitting, parent-child hierarchies, and per-document-type pipelines all carry maintenance cost, and on a small corpus they rarely pay it back. Add complexity only when your eval names the failure it fixes.
What Most Guides Get Wrong
Most chunking guides rank the strategies from naive to advanced and tell you to climb the ladder. That framing is backwards, and it sends teams to semantic chunking before their documents justify it. The ladder treats fixed-size splitting as a beginner mistake to be ashamed of, when it is the correct tool for transcripts and chat logs, and pretending otherwise wastes weeks. The guides also obsess over chunk size as a single global constant, as if your FAQ and your 60-page policy handbook deserve the same number, when the honest answer is per-collection ranges tuned by eval. Worst of all, almost none of them tell you to look at the chunks. I have caught a production pipeline emitting 400 empty chunks a day and nobody knew because no screen ever showed a human what the splitter produced. Fancy strategy selection matters far less than one afternoon of eyeballing output and a test set that tells you the truth.
Conclusion
Chunking is not a preprocessing detail; it is half your retrieval quality. My default path: recursive splitting for plain prose, structure-aware splitting the moment your documents have real headings, 10 to 15 percent sentence-boundary overlap, heading paths prepended before embedding, and parent-child retrieval once the corpus or the eval justifies it. Then build the 40-question test set before you change anything, because the team that measures chunking wins and the team that vibes it rebuilds the system in six months.
