My Retrieval Evaluation Lied to Me Three Times

I built GigBid for a university course (AI-4016): a LangGraph multi-agent copilot that finds freelance jobs, drafts proposals grounded in a freelancer's portfolio, and waits for a human to approve. The retrieval step decides which portfolio evidence the writer sees, so I built a benchmark for it early: 600 synthetic dev jobs across three personas (AI/ML, full-stack, data analyst), 511 of which have gold evidence, scored with Recall@5, MRR and nDCG@10. Everything is synthetic, and I label it that way in the repo.

The benchmark gave me confident numbers that were wrong, three separate times. Each time the retriever was fine and the ruler was broken.

Lie 1: the index and the queries used different embedders

My first V0 (plain dense retrieval) scored far below a random baseline. The index had been built with an offline hash embedder I use for tests, while queries were embedded with bge-small. Nothing crashed, because both sides produced vectors and cosine similarity will happily compare vectors from unrelated spaces. I got a number, and the number was noise.

The fix is boring. The index manifest now stores the embedder name and dimension, and querying with a different embedder raises an error. There is a unit test for it. A silent mismatch is worse than a crash, because a crash gets fixed and a bad number gets tuned around.

Lie 2: chunk ids versus item ids

The chunker splits case studies and repos into pieces, so retrieved ids look like <item>_chunk_00. The gold labels name whole items. My scorer compared them directly, so a correct hit on the right case study was scored as a miss. I only found it by reading rankings by hand next to the scores. The fix was to roll chunk ids up to their parent item before scoring. The general lesson is that whenever the indexing unit (a chunk) differs from the labelling unit (an item), the scorer has to know about both. Otherwise every change to the chunker silently changes your accuracy, and you will blame the retriever for it.

Lie 3: the jobs and their gold labels disagreed

After the first two fixes, V0 still scored 0.056 Recall@5. Random scored 0.162 and BM25 over the skill tags scored 0.096. A semantic retriever should not lose to keyword matching on tags, so I printed sample jobs.

One was titled "Time-Series Demand Forecasting with LightGBM", but its skills field said Docker, OpenAI API, pgvector and Hugging Face. V0's rank-1 result was the LightGBM case study, which was the right answer, and it was scored as wrong. My job generator had sampled title, description and skills independently. Gold evidence was derived from the skills field, so the eval measured agreement with random tags.

Two more problems were hiding behind it. A rule forced every job to have at least one gold item, which gave jobs the persona should skip positive evidence. And my random baseline drew from experience items only, while V0 searched everything, including past proposals, so the comparison was unfair.

The fixes:

• Jobs are generated from archetypes where title, description and skills belong together, and the description mentions 3 to 6 of its skills in prose.
• Jobs with no gold evidence are excluded from retrieval metrics but still count in the bid-or-skip evaluation.
• Random is computed over the same pool as V0.
• Every result file now carries a Recall@5 ceiling, because Recall@5 cannot exceed 5/|gold| when a job has up to 8 gold items. The mean ceiling is 0.6806, and the oracle scores 0.5564. An oracle below 1.0 was not a bug.

What the corrected numbers showed

All figures are from evals/results/ on the 511-query dev set.

Past proposals poisoned the pool. Proposals are 78% of the chunks and are never gold, because they are outcome signal and not evidence. Letting V0 retrieve them dropped Recall@5 from 0.2867 to 0.0519. I now restrict retrieval to evidence types by default and keep the unfiltered run as its own result file.

Plain BM25 nearly matches dense retrieval. BM25 on the job text is at 0.2760 against 0.2867 for V0, and its nDCG@10 is higher (0.4720 against 0.4617). My guess is that my archetype jobs name their skills in prose by construction, which favours keyword matching. That is a guess, not a measurement.

Hybrid gave a small gain. I pre-registered a gain of at least 8 points for hybrid over V0. I got +2.54 points. On the AI/ML persona it was larger, from 0.2222 to 0.3450.

Reranking cost far more than it gave. V3 added +1.87 points of Recall@5 over V2 and +0.0008 nDCG@10, at roughly ten times the p95 latency (296.7 ms to 2,982.4 ms).

Decomposition hurt. Splitting the job into per-requirement queries dropped Recall@5 by 5.91 points and nDCG@10 from 0.4617 to 0.4014, and cost 363,928 extra tokens. I shipped V2 hybrid with persona filtering.

Outcome-aware boosting did nothing measurable on gold Recall@5 (0.3308 to 0.3298).

What I take from this

In all three cases a plausible number was wrong because of plumbing, and I only caught it by reading raw examples next to the scores. Two habits would have saved me the most time: a random baseline over the same pool as the system under test, and a printed ceiling next to every metric.

What the three bugs share is that none of them raised an error. A mismatched embedder, an id that is one suffix off, and a skills field unrelated to its job text all produce a clean run and a number in a results file. Metrics hide this kind of failure well, because a low score looks like "retrieval is hard" and invites tuning the wrong thing. For a stretch I was about to tune prompts and rerank settings against a ruler that was bent.

What worked was boring. I printed three jobs with their titles, skills, gold items and top-5 results side by side, and read them. That took ten minutes and found more than the metric dashboards did. I would now do it before the first benchmark run and not after the third surprise. I would also write the sanity checks first: the embedder guard, an id-rollup test, and a check that a job's skills actually appear in its text.

The limits matter too. The data is synthetic, the jobs are archetype-generated, and gold labels are derived from skill tags, so my numbers measure agreement with that key and not real client intent. Latencies are from my own machine. The calibrated LLM judge and the blind human panel were not run, so I make no quality claims beyond retrieval. The A/B test I ran on proposal styles used a simulator that rewards metric-led openers, so it checks my plumbing and says little about real clients.

The code, results files and the full report are in the repo: github.com/rauf17/gigbid.