Most RAG tutorials get you a working demo in an afternoon: chunk some documents, embed them, stuff the top results into a prompt. That pipeline works beautifully at 50 documents and quietly falls apart somewhere between 5,000 and 50,000 — not because the model got worse, but because retrieval quality did.
Chunking is a product decision, not a default
The default "split every 500 tokens" chunking strategy treats every document like undifferentiated text. A support article, a legal contract and a changelog all fail differently when chunked naively — the fix is chunking that respects each content type's actual structure, not one universal setting.
- Structured docs (contracts, policies) chunk by section heading, never mid-clause — a clause split across two chunks retrieves neither correctly.
- Conversational content (support tickets, chat logs) chunks by exchange, preserving the question-answer pair as one retrievable unit.
- Code and changelogs chunk by logical unit (a function, a version entry) — token-count chunking routinely splits a single relevant example in half.
The retrieval pipeline that held up at scale
Production retrieval, not the demo version
The single highest-leverage change we've made across client RAG systems is adding a reranking step. Vector similarity alone reliably returns results that are topically related but not actually the best answer — a cross-encoder reranker scoring the initial candidates catches that gap, and it's cheap relative to the generation call that follows it.
"The answer isn't in the retrieved chunks" is nearly always a retrieval bug, not a generation bug — teams that debug it by tweaking the prompt are fixing the wrong layer. Log what got retrieved before touching the prompt.
Evaluation is the part that never makes the demo
We run a fixed set of representative queries with known-correct answers against every retrieval change, before it ships. Without that harness, a chunking or reranking tweak that helps one query can silently regress ten others — and you won't find out until a user does.
34%
improvement in answer accuracy after adding a reranking step
3 types
of chunking strategy we run in parallel across different content types
100%
of retrieval changes gated behind a regression eval before shipping
“When a RAG answer is wrong, the prompt is rarely the bug. It's almost always what got retrieved before the model ever saw the question.”
Key Takeaways
- Chunking strategy should match content structure — one universal token-count setting degrades differently across document types.
- Hybrid search plus a reranking step consistently outperforms vector similarity alone, and it's the single highest-leverage addition we've made.
- Debug "the answer isn't in the context" as a retrieval problem first — tuning the prompt fixes the wrong layer almost every time.
- Build a fixed regression eval set before you need one — without it, retrieval tweaks trade one query's accuracy for another's silently.
Bringing it together
RAG that works in a demo and RAG that works at scale differ almost entirely in the retrieval layer, not the generation call. Invest there first — the model was probably never the bottleneck.
