Reranking in RAG: Boost Relevance With Cross-Encoders
Retrieval-augmented generation (RAG) systems live or die by the quality of the chunks they feed into the language model. If your retriever surfaces mediocre passages, no amount of prompt engineering will save the answer. This is where RAG reranking comes in: a second-stage scoring step that reorders your retrieved candidates so the most relevant chunks land at the top before generation.
In this guide we'll walk through why reranking matters, how cross-encoders differ from the bi-encoders used in vector search, and how to add a reranking stage to an existing pipeline without blowing up your latency budget.
Why First-Stage Retrieval Isn't Enough
Most RAG pipelines start with a vector search over embeddings. You embed your query, run an approximate nearest neighbor (ANN) search against a vector store like pgvector, Pinecone, or Qdrant, and pull back the top k chunks. This is fast and scales to millions of documents, but it has a fundamental limitation.
Vector search relies on bi-encoders: the query and each document are embedded independently, then compared with cosine similarity. Because the two texts never "see" each other during encoding, the model can miss subtle relevance signals. A chunk that mentions the right keywords but answers a different question can score deceptively high, while the passage that actually contains the answer sits at rank 8.
The common fix is to retrieve more candidates than you need — say, the top 50 — and hope the right chunk is somewhere in that set. But you can't stuff 50 chunks into a prompt, and lost-in-the-middle effects mean the model often ignores content buried in a long context. You need a smarter way to pick the final handful.
How Cross-Encoders Change the Game
A cross-encoder takes the query and a candidate document together as a single input and outputs a single relevance score. Because the transformer attends across both texts simultaneously, it captures fine-grained interactions — negations, entity matches, and semantic nuance — that a bi-encoder simply cannot.
The tradeoff is cost. A bi-encoder embeds each document once and reuses it forever. A cross-encoder must run a fresh forward pass for every query-document pair at request time. Scoring a million documents this way would be hopeless. That's exactly why cross-encoders are used for reranking a small candidate set rather than for first-stage retrieval.
The Two-Stage Pattern
The winning architecture combines the strengths of both:
Stage 1 — Retrieve: Use a fast bi-encoder vector search (optionally combined with keyword/BM25 search in a hybrid setup) to pull back a broad candidate pool, commonly the top 25 to 100 chunks.
Stage 2 — Rerank: Pass each of those candidates through a cross-encoder alongside the query, sort by the new relevance scores, and keep only the top 3 to 8 for the prompt.
This gives you recall from the cheap first stage and precision from the expensive second stage, applied only to a manageable number of documents.
Adding a Reranker to Your Pipeline
You have two broad paths: run an open-source model yourself, or call a hosted reranking API.
Self-Hosted Open Models
Open-source cross-encoders have matured a lot. Models in the BGE reranker family and Jina's reranker series are strong, permissively licensed options you can run on your own GPUs or CPUs. They integrate cleanly through libraries like Sentence Transformers and Hugging Face Transformers. Self-hosting keeps your data in-house and removes per-call costs, at the price of managing inference infrastructure.
Hosted Reranking APIs
If you'd rather not run models, providers such as Cohere, Voyage AI, and Jina offer dedicated rerank endpoints. You send the query plus a list of candidate texts, and the API returns ranked scores. These are simple to adopt and handle scaling for you, but add network latency and usage fees.
A Minimal Flow
Regardless of provider, the logic is the same:
1. Embed the query and run vector search for the top 50 candidates.
2. Send the query and those 50 chunk texts to the reranker.
3. Sort by returned relevance score.
4. Take the top 5 and build your prompt from them.
Managing Latency and Cost
Reranking adds a step, so budget for it deliberately. A few practical levers:
Tune your candidate count. Reranking 100 chunks is more accurate but slower than reranking 25. Measure retrieval recall first — if the correct chunk is almost always in your top 30, there's no point reranking 100.
Batch the scoring. Cross-encoders parallelize well. Score all candidates in a single batched call rather than looping one at a time.
Cache aggressively. For repeated or templated queries, cache reranked results keyed on the query and candidate set.
Pick a right-sized model. Smaller rerankers with a few hundred million parameters often deliver most of the gain at a fraction of the latency of the largest ones. Test before assuming bigger is better.
Measuring Whether It Actually Helps
Don't add reranking on faith — verify it moves your metrics. Build a small labeled evaluation set of queries with known-relevant chunks, then compare retrieval-only against retrieval-plus-rerank using ranking metrics:
Recall@k tells you whether the right chunk appears in your final top k. Mean Reciprocal Rank (MRR) and nDCG reward putting the best chunk near the top, which matters given lost-in-the-middle effects. Finally, evaluate the end-to-end answer quality, since better ranking should translate into more accurate, less hallucinated responses.
In many real pipelines, reranking is one of the highest-leverage upgrades available — often improving answer relevance more than swapping the generation model itself, and usually for far less engineering effort.
Common Pitfalls to Avoid
Reranking too few candidates. If your first stage only returns 5 chunks and the answer isn't among them, the reranker can't rescue you. Retrieve wide, then narrow.
Ignoring chunk quality. Reranking can't fix badly chunked documents. If your chunks split sentences awkwardly or mix unrelated topics, address chunking first.
Mismatched query formatting. Some rerankers expect specific instruction prefixes or query templates. Follow the model card so you don't quietly degrade scores.
Skipping evaluation. Latency is real and gains vary by domain. Always confirm the improvement with your own data before shipping to production.
When Reranking Is Worth It
Reranking shines when precision matters: customer support bots, legal and compliance search, technical documentation assistants, and any domain where returning the wrong passage has real consequences. If your corpus is small and clean, or your queries are trivially matched, you may not need it. But for the messy, large, real-world knowledge bases most teams deal with in 2026, a cross-encoder reranking stage is one of the most reliable ways to make RAG genuinely trustworthy.
Ready to build real AI skills? Join the September 2026 cohort at Class For Jobs. Explore Advanced AI — a hands-on, live program to build and ship production AI applications, live and instructor-led with career support, resume help, and job-placement assistance.
Related reading
