Thursday AI Tool/Workflow. This week's tool: CrossEncoder from sentence-transformers, used as a reranker. Your retriever (a bi-encoder or BM25) grabs the top-k candidates quickly. Then a cross-encoder like cross-encoder/ms-marco-MiniLM-L6-v2 reads the query and each candidate together, scores how relevant each one is, and you keep only the top-n for the prompt.
It adds one extra model call and a few lines of code, and it targets a common RAG failure: the right chunk was retrieved, but it came in at #3 and the wrong chunk at #1 is the one the LLM ended up leaning on.
Here's how to set it up, with a script I ran on the box, the real output (including a query where the reranker made things worse), CPU timings, and the gotchas I hit.
What it is
A cross-encoder is a transformer that takes a pair of texts as a single input and outputs one relevance score. In sentence-transformers it's the CrossEncoder class, with two methods you'll use:
-
model.predict([(query, passage), ...])returns one score per pair. -
model.rank(query, passages, top_k=n)scores every passage against the query and returns a sorted list of{"corpus_id", "score"}dicts (addreturn_documents=Trueto get the text back as well).
The idea goes back to Nogueira & Cho's 2019 paper Passage Re-ranking with BERT: retrieve candidates with BM25, then feed BERT the query as sentence A and the passage as sentence B, and rank by its relevance score. On the MS MARCO passage task their BERT Large re-ranker reached 36.5 MRR@10 on dev, against 16.7 for untuned BM25 alone, and was the top leaderboard entry at the time (27% relative over the previous state of the art).
cross-encoder/ms-marco-MiniLM-L6-v2 is the small, practical descendant of that idea. Per its Hugging Face model card it's trained on MS MARCO passage ranking, has 22.7M parameters, and scores 74.30 NDCG@10 on TREC DL 19 and 39.01 MRR@10 on MS MARCO dev, at 1,800 docs/sec on a V100 GPU. The 12-layer version scores 74.31 / 39.02 at 960 docs/sec, so the 6-layer model is the usual sweet spot.
Bi-encoder vs cross-encoder: why you need both
The sbert.net docs put the difference simply:
- A bi-encoder (
SentenceTransformer, e.g.all-MiniLM-L6-v2) embeds the query and each passage separately. You embed your corpus once, store the vectors and compare them with cosine similarity. That's what makes it fast enough to search millions of chunks. - A cross-encoder sees the query and the passage at the same time, so attention runs across both texts. The docs say this generally gives better performance than a bi-encoder, but it produces no embedding you can index. Every (query, passage) pair is a fresh forward pass.
The docs give a sense of scale: clustering 10,000 sentences with a cross-encoder means scoring about 50 million pairs (around 65 hours), versus about 5 seconds to embed them with a bi-encoder. So you don't choose one or the other. You chain them: the bi-encoder (or BM25) narrows millions of chunks down to k, and the cross-encoder spends its compute only on those k. The Retrieve & Re-Rank page suggests something like 100 candidates.
1. Install
python -m venv .venv && source .venv/bin/activate pip install torch --index-url https://download.pytorch.org/whl/cpu # CPU-only build, smaller download pip install sentence-transformers I used sentence-transformers 6.3.0 (the current release on PyPI, published November 11, 2026, Apache-2.0, Python 3.10+) with torch 2.15.0+cpu and transformers 5.23.0, on an 8-vCPU Intel Xeon box with no GPU.
2. The workflow: retrieve top-5, rerank to top-3
The corpus is ten passages from a made-up SaaS billing help center. Stage 1 uses all-MiniLM-L6-v2 to retrieve the top 5 passages. Stage 2 uses CrossEncoder.rank to keep the top 3. At the end the script runs a tiny labelled eval and measures how rerank latency grows with k.
# rerank_demo.py - retrieve top-5 with a bi-encoder, rerank to top-3 with a cross-encoder import time import torch from sentence_transformers import SentenceTransformer, CrossEncoder, util docs = [ "Refunds: annual plans cancelled within 30 days of purchase get a full refund. After 30 days, annual plans are not refundable, but you keep access until the end of the billing period.", "Monthly plans can be cancelled at any time from Settings > Billing. Cancellation takes effect at the end of the current month and no partial refunds are issued.", "To change your plan from monthly to annual, open Settings > Billing and choose Switch to annual. The unused part of your monthly fee is credited.", "Invoices are emailed to the account owner on the first day of each billing cycle. You can also download past invoices as PDF from Settings > Billing.", "If a payment fails, we retry the card three times over seven days. After the last failed retry, the workspace is downgraded to the Free plan.", "Annual plans are billed once a year and are 20% cheaper than paying monthly.", "To delete your account, the owner must first cancel any active subscription, then go to Settings > Account > Delete account. Deletion is permanent after 14 days.", "Team admins can add or remove seats at any time. New seats are charged pro rata for the rest of the billing period.", "We accept Visa, Mastercard, American Express and UPI. Bank transfer is available for annual plans above 50 seats.", "Exporting your data: go to Settings > Data > Export to download all projects as a ZIP file. Exports are available for 7 days.", ] queries = [ "What happens if I stop paying?", "Do I lose access immediately when I cancel my yearly subscription?", "I paid for a year two months ago. If I cancel now, do I get my money back?", ] # tiny hand-labelled eval set: query -> id of the doc that actually answers it labelled = { "How do I get a PDF of last month's bill?": 3, "Do new team members cost extra?": 7, "How long until my account is gone for good?": 6, "Will I be charged for the full year if I add people in month 6?": 7, "Is it cheaper to pay yearly?": 5, "Can I get a copy of all my projects before I leave?": 9, "What happens if I stop paying?": 4, "Can I pay without a credit card?": 8, "I want to close my account": 6, "How do I upgrade from monthly to yearly billing?": 2, "Do I lose access immediately when I cancel my yearly subscription?": 0, "Can I get a refund for the rest of the month?": 1, "I paid for a year two months ago. If I cancel now, do I get my money back?": 0, "my card got declined, what happens to my workspace?": 4, } t0 = time.perf_counter() bi = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2", device="cpu") t1 = time.perf_counter() ce = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2", device="cpu") t2 = time.perf_counter() print(f"load bi-encoder: {t1-t0:.2f}s | load cross-encoder: {t2-t1:.2f}s") print("cross-encoder max_seq_length:", ce.max_seq_length) doc_emb = bi.encode(docs, convert_to_tensor=True, normalize_embeddings=True) # warm-up so the first timed call isn't paying one-off costs bi.encode("warm up"); ce.predict([("warm", "up")]) K, N = 5, 3 for q in queries: print("\nQUERY:", q) s = time.perf_counter() q_emb = bi.encode(q, convert_to_tensor=True, normalize_embeddings=True) hits = util.semantic_search(q_emb, doc_emb, top_k=K)[0] retrieve_ms = (time.perf_counter() - s) * 1000 print(f"Stage 1 - bi-encoder top-{K} ({retrieve_ms:.1f} ms):") for r, h in enumerate(hits, 1): print(f" {r}. [doc {h['corpus_id']}] cos={h['score']:.3f} {docs[h['corpus_id']][:70]}...") cand = [docs[h["corpus_id"]] for h in hits] s = time.perf_counter() ranked = ce.rank(q, cand, top_k=N) rerank_ms = (time.perf_counter() - s) * 1000 print(f"Stage 2 - cross-encoder top-{N} ({rerank_ms:.1f} ms):") for r, x in enumerate(ranked, 1): doc_id = hits[x["corpus_id"]]["corpus_id"] # map back to the ORIGINAL doc id prob = torch.sigmoid(torch.tensor(x["score"])).item() print(f" {r}. [doc {doc_id}] logit={x['score']:.2f} sigmoid={prob:.3f} {docs[doc_id][:70]}...") # Top-1 accuracy on the tiny labelled set: bi-encoder alone vs bi-encoder + rerank bi_ok = ce_ok = 0 for q, gold in labelled.items(): q_emb = bi.encode(q, convert_to_tensor=True, normalize_embeddings=True) ids = [h["corpus_id"] for h in util.semantic_search(q_emb, doc_emb, top_k=K)[0]] best = ids[ce.rank(q, [docs[i] for i in ids], top_k=1)[0]["corpus_id"]] bi_ok += ids[0] == gold ce_ok += best == gold print(f"\nTop-1 correct on {len(labelled)} labelled queries: bi-encoder {bi_ok}/{len(labelled)}, after rerank {ce_ok}/{len(labelled)}") # How does rerank latency grow with k? (CPU, batch_size=32, same passages repeated) print("\nRerank latency vs number of candidates (CPU, median of 5 runs):") q = queries[0] for k in (5, 20, 50, 100): cand = (docs * 10)[:k] times = [] for _ in range(5): s = time.perf_counter(); ce.predict([(q, d) for d in cand]); times.append(time.perf_counter() - s) times.sort() print(f" k={k:>3}: {times[2]*1000:.0f} ms") 3. Real output
This is from my second run, once both models were in the local cache (log lines removed):
load bi-encoder: 0.30s | load cross-encoder: 0.19s cross-encoder max_seq_length: 512 QUERY: What happens if I stop paying? Stage 1 - bi-encoder top-5 (3.9 ms): 1. [doc 1] cos=0.504 Monthly plans can be cancelled at any time from Settings > Billing. Ca... 2. [doc 4] cos=0.432 If a payment fails, we retry the card three times over seven days. Aft... 3. [doc 0] cos=0.422 Refunds: annual plans cancelled within 30 days of purchase get a full ... 4. [doc 6] cos=0.316 To delete your account, the owner must first cancel any active subscri... 5. [doc 5] cos=0.278 Annual plans are billed once a year and are 20% cheaper than paying mo... Stage 2 - cross-encoder top-3 (14.3 ms): 1. [doc 4] logit=-2.10 sigmoid=0.109 If a payment fails, we retry the card three times over seven days. Aft... 2. [doc 1] logit=-8.04 sigmoid=0.000 Monthly plans can be cancelled at any time from Settings > Billing. Ca... 3. [doc 0] logit=-8.16 sigmoid=0.000 Refunds: annual plans cancelled within 30 days of purchase get a full ... QUERY: Do I lose access immediately when I cancel my yearly subscription? Stage 1 - bi-encoder top-5 (3.6 ms): 1. [doc 1] cos=0.639 Monthly plans can be cancelled at any time from Settings > Billing. Ca... 2. [doc 6] cos=0.611 To delete your account, the owner must first cancel any active subscri... 3. [doc 0] cos=0.521 Refunds: annual plans cancelled within 30 days of purchase get a full ... 4. [doc 2] cos=0.419 To change your plan from monthly to annual, open Settings > Billing an... 5. [doc 5] cos=0.376 Annual plans are billed once a year and are 20% cheaper than paying mo... Stage 2 - cross-encoder top-3 (13.9 ms): 1. [doc 0] logit=1.65 sigmoid=0.839 Refunds: annual plans cancelled within 30 days of purchase get a full ... 2. [doc 6] logit=-1.51 sigmoid=0.180 To delete your account, the owner must first cancel any active subscri... 3. [doc 1] logit=-3.94 sigmoid=0.019 Monthly plans can be cancelled at any time from Settings > Billing. Ca... QUERY: I paid for a year two months ago. If I cancel now, do I get my money back? Stage 1 - bi-encoder top-5 (4.5 ms): 1. [doc 0] cos=0.615 Refunds: annual plans cancelled within 30 days of purchase get a full ... 2. [doc 1] cos=0.599 Monthly plans can be cancelled at any time from Settings > Billing. Ca... 3. [doc 2] cos=0.383 To change your plan from monthly to annual, open Settings > Billing an... 4. [doc 4] cos=0.380 If a payment fails, we retry the card three times over seven days. Aft... 5. [doc 5] cos=0.351 Annual plans are billed once a year and are 20% cheaper than paying mo... Stage 2 - cross-encoder top-3 (14.5 ms): 1. [doc 1] logit=-5.47 sigmoid=0.004 Monthly plans can be cancelled at any time from Settings > Billing. Ca... 2. [doc 0] logit=-5.82 sigmoid=0.003 Refunds: annual plans cancelled within 30 days of purchase get a full ... 3. [doc 4] logit=-9.50 sigmoid=0.000 If a payment fails, we retry the card three times over seven days. Aft... Top-1 correct on 14 labelled queries: bi-encoder 9/14, after rerank 11/14 Rerank latency vs number of candidates (CPU, median of 5 runs): k= 5: 11 ms k= 20: 29 ms k= 50: 63 ms k=100: 126 ms How to read it:
- "What happens if I stop paying?" The bi-encoder put the cancellation passage first. "Stop paying" sounds a lot like "cancel." The cross-encoder moved the failed payment passage (doc 4) to #1, with a big gap between its score (-2.10) and the next one (-8.04).
- "Do I lose access immediately when I cancel my yearly subscription?" The bi-encoder ranked the monthly-plan passage and the account-deletion passage above the one with the answer. The cross-encoder put doc 0 ("you keep access until the end of the billing period") first, with a clearly positive score (1.65 logit, 0.839 after sigmoid).
- "I paid for a year two months ago..." This is where the reranker got it wrong. The bi-encoder had the right passage (doc 0) at #1, and the cross-encoder swapped it with the monthly-plan passage. Answering correctly means connecting "a year" to "annual" and "two months ago" to "after 30 days," and that's reasoning, not relevance matching. Look at the scores, though: -5.47 and -5.82, both near 0 after sigmoid. The model was effectively saying "nothing here is a confident match."
- Toy eval: on 14 hand-labelled queries, top-1 was correct 9/14 times with the bi-encoder alone and 11/14 after reranking. Reranking fixed three queries and broke one. With 14 queries and 10 passages, that's a sanity check, not a benchmark. Run the same comparison on your own queries.
How to pick k (retrieve) and n (keep)
- k is a recall knob. The reranker can only reorder what stage 1 found. If the right chunk isn't in the top-k, it's gone. Measure stage-1 recall@k on real queries and pick the k where it levels off. The sbert docs use 100 as an example. In this demo, with only ten passages, 5 was enough.
- n is a prompt-budget knob. It's how many chunks go into the LLM context. Three to five reranked chunks is a common starting point. Tune it with your answer-quality eval, not by feel.
- Consider a score floor, not just a fixed n. In my run, the confident answers had positive logits and the wrong top-1 had -5.47. If every candidate scores very low, it's often better to say "I couldn't find that" or escalate than to pass weak chunks to the LLM. Pick the threshold from your own labelled data. These scores are not calibrated across models or domains.
- BM25 works as stage 1 too. The sbert docs list lexical search (e.g. Elasticsearch) as a valid retriever. If you already run hybrid search, rerank the fused list.
Latency trade-offs
Measured on my box (CPU only, predict with the default batch size of 32, median of 5 runs):
| Candidates (k) | Run 1 | Run 2 |
|---|---|---|
| 5 | 22 ms | 11 ms |
| 20 | 53 ms | 29 ms |
| 50 | 86 ms | 63 ms |
| 100 | 171 ms | 126 ms |
Retrieval itself took about 4-6 ms per query with the bi-encoder (query embedding plus cosine search over 10 vectors). Takeaways:
- Rerank cost grows roughly linearly with k, because every candidate is a separate pair. Doubling k roughly doubles stage-2 latency.
- Between two identical runs on the same box, timings moved by up to about 2x. Benchmark on your own hardware, under load, and look at p95, not a single run.
- If 100 candidates on CPU is too slow for your budget, you have three levers: a smaller k, a smaller model (the model card lists L2 and L4 MiniLM variants with higher docs/sec and lower scores), or a GPU. The sbert usage docs also have a "Speeding up Inference" section (the
CrossEncoderconstructor acceptsbackend="onnx"or"openvino"). I didn't benchmark those here.
Gotchas I hit while testing
-
rank()returnscorpus_idrelative to the list you passed in, not your global doc IDs. If you pass the top-5 hits,corpus_idis 0-4. The script maps it back withhits[x["corpus_id"]]["corpus_id"]. Mixing these up gives you a confidently wrong answer and no error. - MS MARCO cross-encoders return logits, not probabilities. My scores ranged from about -11 to +3. The sbert docs say to load with
activation_fn=torch.nn.Sigmoid()if you want 0-1 scores, and that this doesn't change the ranking. I applied the sigmoid manually so the output shows both. - A reranker is not a reasoner. It made the "paid for a year two months ago" query worse. It scores relevance, and a passage that mentions "cancelled... any time... no partial refunds" looks relevant. Keep queries like this in your eval set.
- Cold start is real. The first run downloaded both models: 8.87 s and 7.98 s to load. Once cached, loading took 0.30 s and 0.19 s. Bake the models into your image or warm them up on startup, not on the first user request. The script also does one warm-up call before timing.
- Unauthenticated Hub warning. Every download printed "You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN...". It's harmless, but in CI you'll hit rate limits sooner without a token.
-
max_lengthis deprecated in 6.3.0. Readingce.max_lengthraised aDeprecationWarningtelling me to usemax_seq_length(it's 512 for this model). Query and passage share that 512-token budget, so very long chunks get truncated before they're scored.
Checklist before you ship it
- Pin
sentence-transformers,torchand the exact reranker model in your image, and pre-download the weights. - Measure stage-1 recall@k on 50-100 real queries before picking k.
- Rerank with
CrossEncoder.rank(query, candidates, top_k=n)and mapcorpus_idback to your own chunk IDs. - Log the reranker score for every chunk you send to the LLM.
- Pick a "no good match" threshold from labelled data, if your product can say "I don't know."
- Compare answer quality with and without reranking on the same eval set, and include the reasoning-heavy queries.
- Benchmark stage-2 latency at your real k on production hardware and track p95.
- Re-run the eval whenever you change the retriever, chunking or reranker model.
References
- Sentence Transformers, Cross Encoder usage (
predict,rank, logits vs sigmoid): https://sbert.net/docs/cross_encoder/usage/usage.html - Sentence Transformers, Retrieve & Re-Rank: https://sbert.net/examples/sentence_transformer/applications/retrieve_rerank/README.html
- Sentence Transformers, Cross-Encoders (bi-encoder vs cross-encoder): https://sbert.net/examples/cross_encoder/applications/README.html
- Model card, cross-encoder/ms-marco-MiniLM-L6-v2: https://huggingface.co/cross-encoder/ms-marco-MiniLM-L6-v2
- Model card, sentence-transformers/all-MiniLM-L6-v2: https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2
- sentence-transformers on PyPI: https://pypi.org/project/sentence-transformers/
- Nogueira & Cho (2019), Passage Re-ranking with BERT: https://arxiv.org/abs/1901.04085
Do you rerank before the prompt, or does your top-k go straight to the LLM?
