Chunking — deciding what unit of text gets embedded, retrieved and fed to the model — is the least glamorous decision in a RAG stack and one of the most consequential. Bad chunking poisons everything downstream, and no reranker can fully rescue it.

The naive default — fixed 512-token windows with overlap — embeds boundaries that correspond to nothing: questions split from their answers, tables from their headers, caveats from the claims they modify. Every chunk is then retrieved as an independent truth, and the model stitches contradictions out of fragments. Semantic chunking (split at embedding-similarity drop-offs) helps; structure-aware chunking — respecting markdown headers, code blocks, tables, list membership — helps more, because documents already encode their own semantics in layout.

The state of the art has moved beyond single-level chunks entirely. Parent-child retrieval embeds small, precise chunks but returns their larger parent context on a hit — precision at matching time, coherence at generation time. Hierarchical summariesindex a document at several granularities, letting the query choose the zoom level. And contextual retrieval (Anthropic's 2024 technique) prepends a short LLM-written description situating each chunk in its document before embedding — Anthropic reported it cutting retrieval failures by a third to a half combined with hybrid search; the cost is paid once at indexing, and compounds on every query.

Practical Guardrails

• Chunk on structure first: headers, sections, code fences, table rows — only fall back to size windows where structure is absent.
• Keep provenance in the chunk: document title, section path, and revision ID travel with every chunk; retrieval without citation context breeds confident misuse.
• Tune per modality: code wants function-level units; legal text wants clause-level; transcripts want turn groups with speaker labels. One size fits nothing.
• Evaluate retrieval directly: golden question-to-chunk sets, recall@k and MRR — iterating chunking without retrieval metrics is tuning blindfolded.

The meta-lesson generalizes: in RAG, the pipeline is only as good as the unit of meaning it retrieves. Spend a week on chunking before spending a dollar on a bigger model.