Basic RAG works for demos. Real data needs more.
Hybrid search
Semantic search finds meaning but can miss exact terms such as product codes or names. Keyword search (BM25) finds exact words but misses paraphrases. Hybrid search runs both and merges the results.
Reranking
First retrieve a generous set (say 20 chunks), then use a reranking model to score each against the question and keep the best few. It is slower but often noticeably more accurate.
Query rewriting and expansion
- Rewrite a follow-up like "and for the UK?" into a complete question using the chat history.
- Generate several versions of the question and combine the results (multi-query).
- Ask the model to extract filters, such as country or date, from the question.
Better chunk structure
- Parent-child retrieval: search small chunks, but give the model the larger parent section.
- Contextual retrieval: add a short description of where a chunk comes from before embedding it.
- Metadata filters narrow the search before ranking.
Citations
Number the retrieved chunks, ask the model to cite them, then check in code that every cited number exists. Show the source to the user so they can verify.
Measure retrieval
- Build a test set of questions with the passage that should answer each.
- Recall@k: how often the right chunk is in the top k.
- Precision: how many retrieved chunks were actually useful.
- Also test faithfulness: is the answer supported by the retrieved text?
Change one thing at a time (chunk size, k, reranker) and keep what the numbers prove.