Hybrid Search RAG: Combining BM25, Vector Search and Reranking

Why one retriever is often insufficient

Dense embeddings are excellent at paraphrases and synonyms, but exact IDs, error codes, acronyms, and negation can be difficult. BM25-style lexical retrieval has the opposite profile: it rewards important terms and exact overlap but may miss a semantically equivalent phrase. Enterprise search frequently contains both kinds of queries.

The engineering question is therefore not which retriever is universally best. It is how to combine them while controlling latency, index complexity, and evaluation quality.

A practical hybrid pipeline

A common design runs lexical and dense retrieval in parallel, merges their ranked lists, and then reranks a shortlist with a cross-encoder or another more expensive relevance model. Reciprocal Rank Fusion is useful because the underlying retrievers may produce scores on incompatible scales. Fusion uses rank positions rather than pretending those raw scores are directly comparable.

After fusion, reranking can examine the query and candidate text together. Because the reranker sees only a shortlist, its higher per-document cost is manageable.

• Retrieve a wider candidate set than the final context size.
• Fuse lexical and dense results without relying on incompatible raw scores.
• Rerank the shortlist and pass only the strongest evidence onward.

Where metadata filters belong

Hard constraints such as tenant, permissions, effective date, product, or region should be enforced by the retrieval system, not left to the model. If a user is not allowed to see a document, it should not enter the model context in the first place.

Pre-filtering restricts the search space before similarity ranking. Post-filtering removes disallowed results after retrieval. For security and correctness constraints, pre-filtering is generally the safer design when the index supports it.

How to evaluate hybrid retrieval

Build a query set containing exact-match questions, paraphrased questions, domain terminology, ambiguous questions, and multi-part questions. Label the relevant evidence and compare lexical-only, vector-only, hybrid, and hybrid-plus-reranking variants. Track recall at k, ranking quality, latency, and cost.

Do not evaluate only the final generated answer. If the correct evidence never enters the candidate set, prompt changes cannot repair the retrieval layer.

• Measure retrieval independently before measuring generation.
• Include rare identifiers and domain-specific vocabulary in the test set.
• Track p50 and p95 retrieval latency, not only average latency.

Production considerations

Cache stable embeddings and use incremental indexing. Log query transformations and retrieved IDs so failures are reproducible. When the corpus changes, monitor freshness and deletion propagation. Finally, make the system capable of saying that evidence is insufficient instead of forcing a generation step from weak candidates.