Shipping RAG that survives production

Most RAG demos look convincing because the corpus is small, the questions are written by the same people who indexed the docs, and nobody measures what happens when the answer is not in the index. Production is the opposite of that.

Start with retrieval, not the prompt. Chunk by document structure (headings, tables, API sections) instead of a fixed token window. Hybrid search — lexical plus embeddings — consistently beats embedding-only search on product docs, tickets, and policy text. If you skip reranking, you will spend weeks blaming the model for answers that never had the right passage in context.

Treat citations as a product requirement. Persist the retrieved chunk ids with every response, and refuse to show a confident answer when the top passages score below a threshold. Users forgive “I don’t know” far faster than a fluent hallucination with a fake source.

Evaluations have to run on real questions. Keep a gold set of 50–100 traces from actual users, labeled for groundedness, completeness, and retrieval hit rate. Offline evals catch regressions when you change chunking or models. Online evals — thumbs down, edit distance against a later human answer, escalation to a human — tell you whether the system is still useful.

The failure modes we see after launch are rarely “the model is dumb.” They are stale indexes, permissions leaking into the wrong tenant, tables flattened into unreadable text, and multi-hop questions that need two documents the retriever never joins. Instrument each of those. If you cannot say which stage failed, you cannot fix it.

Ship a thin vertical: one corpus, one user job, one eval dashboard. Then widen. A production RAG system is a retrieval product with a language model on top — not a chatbot with files attached.