I typed ERR_CONN_4012 into my hybrid search and got 30 pages about connection errors in general. The one runbook that actually contained the string ERR_CONN_4012 sat at position 31.
BM25 had ranked it #1. Of course it did: exact token, rare token, short doc. The vector index didn't have it in its top 50 at all. Then Reciprocal Rank Fusion merged the two lists and quietly buried the only correct answer below every vaguely related page.
Nothing was broken. Reciprocal Rank Fusion with k=60 did exactly what the formula says. This post is about that formula, why the textbook default punishes "one retriever is very sure" results, and what I changed.
TL;DR
- Reciprocal Rank Fusion (RRF) scores each doc as
sum(1 / (k + rank))across retrievers. With the common defaultk=60, rank #1 is worth1/61 = 0.0164and rank #50 is worth1/110 = 0.0091. The curve is almost flat. - Any doc ranked 61st or better in both lists beats a doc ranked #1 in only one list. In general, the cutoff is rank
k + 1. - Exact-match queries (error codes, SKUs, function names) are exactly the case where BM25 is right and embeddings miss, so RRF demotes them by design.
- If a reranker only sees the fused top 20, the buried doc never even gets a second chance.
- Fixes: detect identifier queries and pin exact matches, retrieve deeper on the vector side, lower
kor weight BM25, and widen the rerank window.
How does Reciprocal Rank Fusion actually score documents?
Reciprocal Rank Fusion ignores raw scores and uses only rank positions. Each retriever contributes 1 / (k + rank) for every doc it returned, and a doc missing from a list contributes zero.
def rrf(rankings: list[list[str]], k: int = 60) -> dict[str, float]: scores: dict[str, float] = {} for ranking in rankings: for rank, doc_id in enumerate(ranking, start=1): scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank) return dict(sorted(scores.items(), key=lambda x: -x[1])) That's the whole algorithm. It's popular for good reasons: BM25 scores and cosine similarities live on completely different scales, and RRF sidesteps calibration entirely. No normalization, no tuning, works out of the box. Elasticsearch, OpenSearch, Weaviate, Qdrant and most RAG tutorials reach for it, and k=60 is the number almost everyone copies from the original 2009 paper.
The trouble is what k=60 does to the shape of the curve.
Why does RRF with k=60 bury exact keyword matches?
RRF with k=60 makes appearing in two lists worth far more than ranking first in one. Here are the actual numbers:
| Situation | RRF score |
|---|---|
| #1 in BM25, absent from vector list | 1/61 = 0.01639 |
| #50 in both lists | 2/110 = 0.01818 |
| #61 in both lists | 2/121 = 0.01653 |
| #62 in both lists | 2/122 = 0.01639 (tie) |
Read the second row again. A document that both retrievers considered mediocre, 50th place each, outscores the document BM25 was most confident about.
The break-even is easy to derive. A doc at rank r in both lists beats an exclusive #1 when:
2 / (k + r) > 1 / (k + 1) k + r < 2k + 2 r < k + 2 So with k=60, every doc ranked 61st or better in both lists outranks your exclusive #1. If each retriever returns 50 results, that means every single overlapping doc wins.
That's where position 31 came from. My query hit 30 generic "connection error / timeout / retry" pages that both BM25 and the embedding model liked. All 30 appeared in both top-50 lists. The runbook appeared in one. Final position: overlap count plus one. It's not bad luck, it's arithmetic.
Why do embeddings miss error codes in the first place?
Embedding models compress meaning, and an identifier like ERR_CONN_4012 has very little meaning to compress. The tokenizer chops it into pieces like ERR, _, CONN, _, 401, 2, and the model maps the whole thing to roughly "some connection error." Every page about connection errors lands nearby.
BM25 sees the opposite picture. The full token is rare across the corpus, so its IDF is huge, and the one doc containing it shoots to the top.
This split is the entire reason hybrid search exists. The queries where one retriever is decisively right and the other is lost are the queries hybrid search was supposed to save. Flat RRF treats that disagreement as noise and votes it down.
Does a reranker fix bad RRF fusion?
A cross-encoder reranker only fixes RRF if the correct doc is inside the window it reranks. A lot of pipelines fuse, take the top 20, and rerank those. My runbook was at 31, so the reranker never saw it. The most accurate component in the pipeline was handed a list that didn't contain the answer.
If you rerank, the window has to be at least as large as the plausible overlap between your retrievers. With two top-50 lists, the worst case for an exclusive #1 is position 51. I rerank the fused top 60 now. It costs more cross-encoder calls, but a reranker that sees the answer is cheaper than one that confidently orders 20 wrong ones.
How do you fix Reciprocal Rank Fusion for exact-match queries?
Four changes, roughly in order of impact.
1. Detect identifier queries and pin exact matches
If the query contains something that looks like a code, the user wants that string. Don't make them win a popularity contest for it.
import re IDENT = re.compile(r"\b(?:[A-Z]{2,}[_-][A-Z0-9_-]+|[A-Z]+-\d+|0x[0-9a-fA-F]+|\w+\(\))") def hybrid_search(query, bm25, vec, k=60, n=50): bm = bm25.search(query, n) vc = vec.search(query, n) fused = list(rrf([bm, vc], k=k)) idents = IDENT.findall(query) if idents: exact = [d for d in bm[:5] if any(i in doc_text(d) for i in idents)] fused = exact + [d for d in fused if d not in exact] return fused The check against doc_text matters. BM25's top 5 can still contain partial matches on ERR or CONN, and you only want to pin docs that contain the literal string.
2. Retrieve deeper on the vector side
An exclusive #1 loses because the other list gives it zero. Give the other list a chance to give it something. If the vector search returns 500 instead of 50 and the runbook shows up at rank 400, it earns 1/460 = 0.00217 extra:
1/61 + 1/460 = 0.01857 (runbook) 2/110 = 0.01818 (doc at #50 in both) Now it beats every doc ranked 50th in both. It still loses to shared docs near the top, but it climbs out of the basement. Deeper ANN retrieval is cheap compared to the LLM call that follows.
3. Lower k, or weight the retrievers
Lower k makes top ranks matter more. From the break-even above, an exclusive #1 is beaten by docs in both lists down to rank k + 1:
| k | Exclusive #1 loses to shared docs ranked... |
|---|---|
| 60 | 1 through 61 |
| 20 | 1 through 21 |
| 10 | 1 through 11 |
| 1 | 1 through 2 |
Weighted RRF multiplies each retriever's contribution. Be careful: the weight also boosts shared docs' BM25 half, so it's weaker than it looks. With BM25 weight 2 and k=60, a shared doc at rank r scores 3/(60+r) against the exclusive doc's 2/61. Shared docs still win down to rank 31. Weight 3 still lets shared docs win down to rank 21. Weights help, but they don't solve the identifier case alone.
I ended up with k=20 as the default and kept the identifier pin as the real fix.
4. Look at disagreement, don't average it
The most useful debugging change was logging, per query, how many docs appeared in only one list and where they landed after fusion. A doc that's #1 in one retriever and missing from the other is a signal, not an outlier. If that pattern shows up on a large share of your queries, your two retrievers are answering different questions and flat fusion is hiding it.
When is RRF k=60 still the right choice?
RRF with k=60 is a fine default when your queries are natural-language questions and both retrievers mostly agree. In that regime, the flat curve smooths out noise from either retriever, and consensus really is a good proxy for relevance. It's also hard to beat for zero-tuning setups.
It's the wrong default for corpora full of identifiers: runbooks, API references, ticket histories, log search, product catalogs, legal citations. There, the query often has exactly one correct answer, and only the lexical retriever knows it.
So why did my #1 BM25 match fall to position 31?
Reciprocal Rank Fusion with k=60 gives rank #1 a score of 1/61 and rank #50 a score of 1/110, so the rank curve is nearly flat and showing up in two lists dominates ranking high in one. Mathematically, any document ranked k + 1 or better in both lists beats a document ranked #1 in only one. My runbook was BM25's #1 and absent from the vector top 50, so it landed behind all 30 documents that appeared in both lists. To fix it, detect identifier-style queries and pin literal matches, retrieve deeper on the vector side so exclusive hits earn partial credit, lower k (I use 20), and rerank a window wide enough to include the worst-case position of an exclusive #1.
Written by the developer behind Preterview, an interview prep platform.


