What a deeper cross-validation audit revealed about reproducibility, evaluation composition, and the limits of machine-learning performance metrics.
There is a particular kind of confidence that comes from running an experiment twice and getting exactly the same answer.
Your code executes. The metrics match. The predictions agree. Nothing crashes, and the outputs are reproducible.
It is tempting to consider that a successful experiment.
But reproducibility and validity answer different questions.
I recently spent time investigating this distinction in QueleaGuard, a machine-learning research project studying the environmental and spatial factors associated with Red-billed Quelea occurrence around the Ahero Rice Irrigation Scheme in Kisumu County, Kenya.
The project had already established a spatial cross-validation design and produced a baseline logistic regression evaluation. Rather than immediately moving to more sophisticated models, I decided to investigate whether the existing evaluation could be independently reconstructed from the committed data and code, and what its results actually represented.
The exercise became a much larger evaluation audit.
01 — The Experiment We Needed to Reproduce
The baseline used logistic regression to distinguish recorded quelea presences from an approximate target-group background sample.
The modelling dataset contained 266 records:
Dataset component| Records
Confirmed occurrence records| 133
Background records| 133
Total| 266
Evaluation folds| 5
The model used ten numerical predictors and three seasonal indicator variables.
The numerical predictors included rainfall, temperature, dewpoint, wind, vegetation index, elevation, slope, distance to water, and rice-related landscape variables.
Each fold was held out once. The remaining four folds were used to fit the model, with numerical predictors standardised using the training data.
The original evaluation produced:
Metric| Mean across five folds
F1-score| 0.593
ROC-AUC| 0.854
Those numbers were interesting, but they were not sufficient evidence that the model generalised reliably.
I wanted to establish something more fundamental first:
«Could I reconstruct the reported experiment using the committed inputs and implementation, without changing anything to make the numbers agree?»
02 — Building a Reproduction Gate
I created a separate audit stage, M0, implemented in "src/cv_audit_m0.py".
The objective was not to improve the model. It was to reconstruct the original baseline and compare its outputs against the previously committed results.
Before fitting anything, the script checked the integrity of the experiment.
It verified:
- The exact versions of the three gated dataset inputs.
- The committed cross-validation protocol.
- The expected 29-column modelling dataset.
- All 266 records and their unique identifiers.
- The 133/133 class distribution.
- The exact fold assignments and per-fold class counts.
- The relationship between background records and the final pseudo-absence dataset.
- The relationship between presence records and the occurrence dataset.
The checks were deliberately strict.
If a critical input differed, the audit was supposed to stop rather than silently continue with a different experiment.
The model was then reconstructed using the same logistic regression specification, including the original feature construction, scaling procedure, five folds and classification threshold.
I also ran the reconstruction twice within the same execution.
The purpose was to distinguish agreement with the historical results from agreement between repeated executions.
03 — The Baseline Reproduced Successfully
The reproduction gate passed all 51 assertions.
The largest absolute differences between the reconstructed and original metrics were:
Metric| Maximum absolute difference
Precision| 0
Recall| Approximately 1.11 × 10⁻¹⁶
F1-score| 0
ROC-AUC| Approximately 1.11 × 10⁻¹⁶
The maximum difference between the two independently executed sets of predicted probabilities was also zero.
No convergence warnings were recorded.
The audit additionally checked the confusion matrices, calibration diagnostics and the composition of a particularly difficult evaluation fold.
This was a useful result.
Reproducibility established.The committed implementation and data could regenerate the historical baseline results to floating-point precision in the recorded environment.
But it did not establish that the baseline was a good model.
It did not establish that its probabilities were calibrated, that the spatial validation design was adequate, or that its performance would transfer to genuinely new locations.
Reproducing an experiment confirms that you can regenerate its outputs. It does not independently confirm that the experiment measures what you intended it to measure.
04 — Looking Beyond the Fold Averages
This distinction motivated the next audit stage, R.
Instead of fitting another model, I kept the reproduced out-of-fold predictions fixed and investigated the composition of their evaluation results.
The question changed from:
How well did the model perform in each fold?
to:
Which observations and locations contributed to those scores?
The analysis examined individual presence-bearing grid cells, decomposed focal locations from the remaining observations, tested the sensitivity of evaluation metrics to removing individual cells, and investigated residual spatial patterns.
Two folds deserved particular attention.
4.1 — Fold 3: A Strong Score Concentrated Around One Location
Fold 3 contained 58 records, including 36 presences.
Its baseline results were:
Metric| Result
Precision| 0.882
Recall| 0.833
F1-score| 0.857
ROC-AUC| 0.886
On its own, that looks encouraging.
However, one location, "cell_0156", contributed 27 of the 36 presence records.
It also accounted for 27 of the fold's 30 true positives.
When the audit separated that focal cell from the remaining evaluation records, the remaining subset contained only nine presences and 22 background records.
Its results were substantially different:
Metric| Remaining subset
Recall| 0.333
Precision| 0.429
F1-score| 0.375
ROC-AUC| 0.667
This does not establish that the model learned that location, nor does it prove that the model would fail elsewhere.
27 of 36 presence records came from one grid cell.The full-fold performance concealed a meaningful difference between the focal location and the remaining evaluation subset.
The remaining sample is small, and the analysis changes the evaluation set rather than refitting the model.
It does, however, demonstrate why a single fold-level score can conceal important differences in evaluation composition.
4.2 — Fold 4: A Different Pattern, With a Similar Problem
Fold 4 contained 113 records, including 69 presences.
Its baseline results were:
Metric| Result
Precision| 0.833
Recall| 0.362
F1-score| 0.505
ROC-AUC| 0.839
Here, "cell_0080" contributed 60 of the 69 presence records.
The focal location accounted for 39 of the fold's 44 false negatives.
Removing that location from the evaluation subset left 51 records, including nine presences and 42 background records.
The remaining subset had:
Metric| Remaining subset
Recall| 0.444
Precision| 0.500
F1-score| 0.471
ROC-AUC| 0.918
Interestingly, the remaining subset's ROC-AUC was higher than the full-fold ROC-AUC, even though its F1-score was slightly lower.
This is a useful reminder that different evaluation metrics describe different aspects of model behaviour.
A strong ranking metric does not necessarily imply strong classification performance at a particular decision threshold.
The subset also contained only two other presence-bearing cells, so its metrics must be interpreted cautiously.
05 — Why I Did Not Simply Remove the Difficult Locations
There is an important methodological distinction here.
The R-stage analysis removed locations from the evaluation set while keeping the original model predictions fixed.
It did not retrain the model without those locations.
Consequently, it tells us how much the composition of an evaluation subset affects the reported metrics.
It does not tell us what would happen if the model were trained without the dominant locations.
That is a separate experiment.
This distinction is particularly important in spatial machine learning, where the same geographic locations can contribute many observations.
Counting records alone can make an evaluation look more informative than it really is.
Sixty observations from two coordinates do not provide the same geographic coverage as sixty observations distributed across sixty independent locations.
The number of records, the number of distinct coordinates, the number of dates and the number of independent spatial sites all deserve attention.
06 — Reproducibility Needs an Audit Trail
Another part of this work was making the audit itself reproducible.
The shared implementation in "src/cv_audit_common.py" records:
- Git commit information.
- Input and output file hashes.
- Python and library versions.
- Model parameters and analysis settings.
- Individual assertions and their results.
- Execution timestamps.
- Changes to protected inputs and existing audit artefacts.
Outputs are written into unique, timestamped directories rather than silently overwriting earlier results.
This matters because an evaluation is not just a number in a report.
It is the result of a particular dataset, implementation, configuration and execution environment.
If any of those change, the relationship between the new result and the old one needs to be explicit.
The project's decision log also records what an analysis establishes, what it does not establish, and which questions remain open.
I find this especially valuable in research engineering:
«Methodological decisions should be traceable, not reconstructed from memory after something goes wrong.»
07 — What Comes Next
The next stage of the audit moves beyond keeping predictions fixed.
The newly committed M1 analysis investigates the influence of training composition through controlled refits, including exact-repeat and cell-balanced weighting variants and reference ablations.
This is deliberately separate from the R-stage evaluation-composition analysis.
Only after those investigations can we make stronger statements about the baseline's sensitivity to the dominant spatial clusters.
Even then, QueleaGuard remains an active research project.
It has not established a validated early-warning system, and its current occurrence modelling should not be interpreted as confirmed infestation or crop-damage prediction.
That distinction matters because the eventual goal is practical: better information about where and when quelea activity may occur, so that agricultural interventions can be planned more intelligently.
Getting there requires more than a model with an attractive ROC-AUC.
08 — The Lesson
I started this exercise wanting to establish whether I could reproduce an existing machine-learning experiment.
I ended up with a much clearer understanding of what its reported metrics did and did not tell me.
There are at least three separate questions worth asking about an ML experiment:
- Can I reproduce it?
Can the committed data, code and configuration regenerate the reported outputs?
- Does the evaluation represent the problem?
Are the test observations sufficiently representative, and do the reported metrics conceal important differences in evaluation composition?
- Does it generalise?
Does the model perform reliably beyond the particular observations, locations and conditions represented in the experiment?
Passing the first question does not automatically answer the other two.
For me, this is part of what separates building an ML model from conducting credible machine-learning research.
The objective is not to produce the most impressive metric.
It is to understand exactly what that metric supports, what it leaves unresolved, and what evidence is needed next.
---
A reproducible result is a starting point for scientific confidence, not the end of the investigation.
Project and implementation
Repository: "QueleaGuard — GitHub" (https://github.com/benardabuto081/QueleaGuard)
Relevant implementation:
- "M0 — Baseline reproduction audit" (https://github.com/benardabuto081/QueleaGuard/blob/main/src/cv_audit_m0.py)
- "R — Evaluation-composition audit" (https://github.com/benardabuto081/QueleaGuard/blob/main/src/cv_audit_r.py)
- "Shared audit infrastructure" (https://github.com/benardabuto081/QueleaGuard/blob/main/src/cv_audit_common.py)
- "Research decision log" (https://github.com/benardabuto081/QueleaGuard/blob/main/docs/assumptions_and_decision_log.md)