Hybrid Search for RAG: Combine BM25 and Vectors
Pure semantic search feels like magic until it isn't. You embed a user's question, run a nearest-neighbor lookup against your vector store, and get back passages that are topically related but miss the exact product code, error string, or acronym the user typed. That failure mode is common enough that many production RAG systems now default to hybrid search: combining sparse keyword retrieval (BM25) with dense vector retrieval to get the best of both worlds.
This post explains why hybrid search RAG consistently outperforms pure semantic search on accuracy, how the two retrieval methods complement each other, and how to implement and tune a hybrid pipeline that you can ship.
Why pure vector search falls short
Dense embeddings map text into a continuous vector space where semantic similarity becomes geometric proximity. This is powerful for paraphrase and concept matching—"how do I cancel my subscription" retrieves a passage titled "ending your membership" even with zero shared words.
But embeddings compress meaning, and compression loses precision. Vector search tends to struggle with:
- Exact identifiers: SKUs, error codes like
ERR_5021, API method names, version numbers, and legal clause references. - Rare or out-of-vocabulary terms: internal jargon, product names, and abbreviations the embedding model never saw during training.
- Long-tail keywords: uncommon words that carry high information but get averaged away in a dense vector.
In these cases the "most similar" vector may be a semantically adjacent but factually wrong passage—and the LLM will confidently ground its answer in it.
What BM25 brings back
BM25 is a sparse, lexical ranking function that scores documents by term frequency and inverse document frequency, with saturation and length normalization. It has been a search staple for decades because it is fast, interpretable, and requires no training. For RAG, BM25 excels precisely where vectors fail: it rewards exact token overlap.
If a user searches for ERR_5021, BM25 will surface the one document containing that exact string, even if it appears in only a handful of documents across your corpus. It is also robust to domain shift—you can drop it onto proprietary data without fine-tuning anything.
The trade-off is that BM25 is literal. It does not understand synonyms, and it fails when the user's vocabulary differs from the document's. That is exactly the gap dense retrieval fills.
Why the combination wins
Keyword and vector retrieval have complementary failure modes. When one misses, the other frequently catches the relevant passage. Fusing their results gives you both lexical precision and semantic recall, which is why hybrid setups reliably beat either method alone on real-world retrieval benchmarks and internal evaluations.
Think of it as an ensemble: you are not choosing between recall and precision, you are hedging across two retrieval strategies that make different kinds of mistakes.
How to fuse the two result sets
The core engineering question is how to merge two ranked lists that produce scores on completely different scales. BM25 scores are unbounded positive numbers; cosine similarities sit between -1 and 1. You cannot just add them. Two approaches dominate.
Reciprocal Rank Fusion (RRF)
RRF ignores raw scores entirely and uses rank position. For each document, you sum 1 / (k + rank) across every result list it appears in, where k is a constant (60 is the common default). Documents that rank highly in either list bubble to the top; documents that rank well in both dominate.
RRF is the pragmatic favorite because it is scale-free, tuning-light, and works out of the box. Elasticsearch, OpenSearch, and several vector databases ship native RRF support in their 2026 releases.
Weighted score normalization
Alternatively, normalize each score set (min-max or z-score) into a common range, then compute a weighted sum: final = alpha * norm_vector + (1 - alpha) * norm_bm25. This gives you an alpha knob to bias toward semantic or lexical matching. It is more powerful but requires per-corpus tuning and is sensitive to outliers, so normalize carefully.
A practical implementation pattern
A typical hybrid RAG retrieval flow looks like this:
- Index once, two ways. Store each chunk with both an inverted index (for BM25) and its embedding (for vector search). Modern engines like OpenSearch, Elasticsearch, Weaviate, Qdrant, and pgvector-backed Postgres can hold both.
- Retrieve in parallel. Run the BM25 query and the vector query at the same time, each returning a top-k (say 20–50 candidates).
- Fuse. Merge with RRF or weighted normalization to produce a single ranked list.
- Rerank. Pass the top fused candidates through a cross-encoder reranker for a final precision boost before sending the top 3–8 chunks to the LLM.
That final rerank step matters. A cross-encoder scores the query and each candidate jointly rather than independently, catching subtle relevance signals that neither retriever can. Hybrid retrieval widens the funnel; reranking sharpens it.
Tuning tips that actually move accuracy
Chunk thoughtfully. BM25 and embeddings react differently to chunk size. Very long chunks dilute BM25 term weighting and blur embeddings. Chunks of roughly 200–500 tokens with modest overlap are a reasonable starting point—then measure.
Retrieve more candidates than you keep. Pull a generous top-k from each retriever before fusion. Fusion and reranking can only choose from what they see, so give them a larger pool.
Preprocess consistently. Lowercasing, tokenization, and handling of code-like tokens affect BM25 heavily. Make sure identifiers such as ERR_5021 are not split apart by your analyzer.
Build an evaluation set. You cannot tune what you cannot measure. Assemble a set of representative queries with known correct passages and track metrics like recall@k, MRR, and nDCG. Compare pure vector, pure BM25, and hybrid so you can prove the lift rather than assume it.
When hybrid may be overkill
Hybrid search adds infrastructure and latency. If your corpus is small, your queries are conversational and paraphrase-heavy, and exact identifiers rarely appear, pure vector search may be good enough. But for enterprise knowledge bases, technical documentation, support systems, legal and financial text, or anything dense with codes and names, hybrid search is usually worth the extra complexity.
The good news for developers and data engineers in 2026 is that hybrid retrieval is increasingly a configuration flag rather than a custom build. The concepts, though, are what let you diagnose why a retrieval is failing—and choosing the right fusion and reranking strategy is where the real accuracy gains live.
Ready to build real AI skills? Join the September 2026 cohort at Class For Jobs. Explore Advanced AI — a hands-on, live program to build and ship production AI applications, live and instructor-led with career support, resume help, and job-placement assistance.
Related reading
