I found the optimal top-k for my RAG system. Then I ran it again.

I built a small retrieval-augmented generation system over a corpus of research papers and wanted to answer an ordinary question: how many chunks should I retrieve? So I did what the tutorials suggest — swept top-k across 3, 5, 8 and 12, scored each configuration with RAGAS, and read off the winner.

The answer was clean. k=8 led on all four metrics. Context recall was 0.53 at k=3, peaked at 0.83 at k=8, and fell back to 0.64 at k=12 — a textbook inverted U, with a story to match. Too few chunks and the generator starves; too many and distractors dilute the prompt. I had a number, a mechanism, and a plot.

Then I ran k=8 twice more, changing nothing.

Recall ranged from 0.611 to 0.833 across three runs of an identical configuration. The 0.833 that made k=8 look like a peak was the high draw of three.

Set the variation within one configuration against the differences between configurations:

On every metric, one configuration varies more against itself than the configurations vary against each other. k=5, k=8 and k=12 are not distinguishable on this evidence. The only ordering that survives is that k=3 under-retrieves — it has the lowest recall in every run.

The inverted U was an artifact of running each configuration once.

The judge moves the numbers more than the parameter does

While building the harness I had changed judge models — the free tier I started on ran out of quota. That turned out to be a second experiment I hadn't meant to run.

Same corpus, same generator, same k=5. Only the judge differs:

Changing the judge moved faithfulness by 0.26 and recall by 0.31. Changing top-k from 3 to 8 — the parameter I was actually studying — moved faithfulness by 0.12.

Part of that gap is different grading. Part is something subtler: the two judges fail differently. One of them intermittently returned output the parser couldn't read, so those samples scored as missing and dropped out of the average. The two numbers were therefore computed over different subsets of questions. Both effects are properties of the judge, and both are invisible in a reported score.

I want to be careful here: I have one run per judge, so I can't cleanly separate a judge effect from an unlucky draw. The honest statement is that the judge shift is about the same size as the run-to-run variation I measured, which is already enough to make absolute RAGAS numbers meaningless without naming the judge that produced them.

The significance test couldn't have found anything

Comparing configuration means throws away the pairing, so I did it properly: per-question differences between two configurations, Wilcoxon signed-rank, each question acting as its own control. Nothing reached significance. For a while I read that as "no difference."

It wasn't. Wilcoxon discards tied pairs, and with n non-zero pairs the smallest two-sided p-value it can return is 2^(1−n):

Below six non-zero pairs, significance at α = 0.05 is arithmetically impossible regardless of the data.

My context-precision comparison had three non-zero pairs and returned p = 0.250. My recall comparison had one and returned p = 1.000. Both are exactly the floor — every non-zero pair pointed the same way, the most extreme result obtainable, and it still couldn't clear the threshold.

My sample collapsed twice over: 18 questions, then 8–9 matched pairs after judge failures, then 1–3 non-zero pairs after ties. Each stage looked survivable. Together they left a test with no power at all.

The tie rate is the more interesting half, though. Eight of nine pairs scored identically between k=8 and k=12. On most questions the chunks ranked 9 through 12 changed nothing the judge could see. That's a finding about retrieval, and it explains the null better than low power does.

The first thing my harness found was a bug in my own corpus

Before any of this, the eval surfaced something I hadn't gone looking for. My "21-paper corpus" was 15 papers. Each one's PDF and its metadata record were being indexed as separate works, so retrieving both counted as retrieving two relevant documents. Every precision number I'd computed was inflated.

Deduplicating on a canonical work ID fixed it. But I'd been quoting 21 for weeks, and nothing except an evaluation harness would have caught it.

What I'd do differently, and what I'd keep

Replicate before believing. At minimum three runs per configuration. If within-configuration spread approaches the between-configuration differences, report that and stop — it's the finding.

Report the judge and generator with every metric. A RAGAS score without them isn't comparable to anyone else's, or to your own from last month.

Check the power before spending the quota. Computing 2^(1−n) is one line and it tells you in advance whether a comparison can possibly reach significance. Mine couldn't, and I found out afterwards.

Report ties and sample attrition. "Not significant" and "no measurable difference" are different claims, and the gap between them lives in the counts that usually go unreported.

Separate what a judge grades from what code can check. My suite has two tracks. Queries about corpus statistics and listings return zero chunks by design, so context-based metrics are undefined for them; those get deterministic assertions instead. Those assertions, and the intent-routing accuracy, came back at 100% across every run. Code-driven checks carried no variance at all. The LLM-judged metrics carried all of it. That contrast is worth designing for.

Limits

This is one system, one 15-paper corpus, 18 scored content questions, and two judges. I'm not claiming a general law about RAGAS or about LLM-as-judge evaluation — I'm claiming that the standard procedure, run on a realistic small setup, produced a confident answer that replication dissolved, and that the failure modes were all invisible from the reported numbers.

If your evaluation is larger, it may be fine. If it's about this size, and you ran each configuration once, you have the result I had before I ran it again.

Harness, golden set, ablation and results: github.com/huali10044/research-rag. The system itself is live at huali-research-rag.streamlit.app.