Retrieval that actually works: engineering notes on production RAG

Teams spend weeks tuning chunk size and overlap, then wonder why answer quality barely moves. In our own evaluation work, the biggest wins almost never came from chunking parameters.

The first win is document hygiene. Duplicated policy documents, three versions of the same handbook, and exported PDFs with broken tables poison retrieval more than any embedding choice. Deduplicate and mark a single source of truth before indexing anything.

The second win is hybrid retrieval. Dense vectors are excellent at paraphrase and weak at exact identifiers. Combining semantic search with keyword search and reranking the merged candidate set consistently outperformed either method alone in our tests.

The third win is asking the model to abstain. An agent that says it could not find something in the handbook and escalates is more valuable than one that produces a fluent, confident, wrong answer. Make abstention an explicit and rewarded behaviour in both instructions and evaluations.

Finally, measure retrieval separately from generation. If you only score final answers, you cannot tell whether the model reasoned badly or was handed the wrong passages.