Chunking Strategies for RAG: A Practical Guide

A retriever can only return what was indexed. If a chunk is too small, it loses the surrounding context that makes it answerable. Too large, and the embedding becomes a blurry average — the relevant span gets drowned out by adjacent noise, and your top-k results stop being useful. The sweet spot depends on the corpus, the embedding model's context window, and the questions you expect users to ask.

Split text into windows of N tokens (commonly 256–1024) with an overlap of 10–20%. It's the baseline for a reason: predictable, fast, trivially parallelizable, and good enough for clean, homogeneous corpora. Use it as your control when evaluating fancier strategies — if a recursive or semantic splitter doesn't beat fixed-size on your eval set, it's not earning its complexity.

Split on a priority list of separators — paragraphs, then sentences, then words — falling back only when a chunk exceeds the size budget. This preserves natural boundaries far better than fixed windows and is the right default for prose: documentation, articles, transcripts, policy text. LangChain's RecursiveCharacterTextSplitter popularized the pattern.

For Markdown, HTML, code, or PDFs with reliable structure, split on the structure itself: headings, sections, function definitions, table rows. Carry the parent heading into the chunk text or metadata so a section about "Refund eligibility" still retrieves on "refund" even when the body never repeats the heading. This is the single biggest quality win on technical docs.

Embed each sentence, then merge adjacent sentences while their cosine similarity stays above a threshold; cut when the topic shifts. The result is variable-length chunks that align with meaning rather than character count. It's expensive at index time and sensitive to the threshold, but on noisy transcripts or long-form essays it consistently beats fixed-size.

Two patterns worth knowing. Late chunking embeds the full document with a long-context encoder, then pools per-chunk vectors from the contextualized token embeddings — each chunk's vector "remembers" the surrounding document. Small-to-big indexes small, focused chunks for retrieval precision but returns the larger parent chunk to the LLM for grounding context. Both decouple the unit you search from the unit you generate against.

With a flat FAISS index, chunk count drives memory and latency linearly — halving chunk size doubles the index. Move to IVF or HNSW once you cross a few million vectors, and re-tune your chunk size against the recall@k of the approximate index, not the flat one you prototyped on. Always attach rich metadata (source, section path, timestamps) so you can pre-filter before ANN search and re-rank after.

1. Build a small eval set of 50–100 real questions with known-good answers.
2. Ship fixed-size 512/64 as the baseline and measure recall@k and answer faithfulness.
3. Layer in structural splitting if the corpus has headings. Re-measure.
4. Try recursive or semantic splitting only if the baseline misses on questions whose answers straddle chunk boundaries.
5. Reserve late chunking and small-to-big for cases where the simpler options have plateaued.

Chunking is an engineering decision, not a default. Treat it like indexing strategy in a database: measurable, tunable, and worth re-visiting whenever the corpus or the questions change. The best RAG systems I've shipped spent more time on the splitter than the prompt.