We trained a 360M parameter code model from scratch and published 26.8% pass@1 on HumanEval.
Two days later a reader asked whether we had checked our instruction data for benchmark overlap. We had not. So we did, and the honest number is 22.6%.
Here is what was in the data.
The scan
Two checks against each instruction set:
- Problem text. The first 80 characters of each HumanEval docstring, whitespace normalized.
- Canonical solutions. Half or more of a solution's 13-token n-grams, with comments and docstrings stripped.
| Dataset | Examples | Docstrings found | Solutions found |
|---|---|---|---|
| evol-codealpaca-v1 (we used this) | 111,272 | 60 problems, 156 examples | 8 problems, 67 examples |
| Magicoder-Evol-Instruct-110K (the "decontaminated" copy) | 111,183 | 52 problems, 116 examples | 8 problems, 66 examples |
| Magicoder-OSS-Instruct-75K (we used this) | 75,197 | 0 | 1 problem, 1 example |
The filtered redistribution differs from the original by 89 rows and still contains 52 of the 60 contaminated HumanEval problems. Switching to it would not have saved us.
Why the filter missed them
The matches are not copies. They are benchmark problems in practice-exam form: reworded, extended with an extra requirement, or ported to Java, each with a full solution.
HumanEval/43, pairs_sum_to_zero, appears with its original docstring, its doctests and a working solution, under an added instruction to make it linear time.
That is not an accident of collection. Evol-Instruct grows a dataset by repeatedly rewriting seed problems, and some of the seeds were benchmark problems. An exact-match filter cannot catch a rewrite, and rewriting is the entire method.
Was the model reciting answers?
No, and this is the part that surprised us.
We compared every chat-model pass against its canonical solution with a difflib ratio on whitespace-normalized lines. A recited answer scores above 0.9. Of 44 passes, the highest was 0.73, on a problem whose canonical solution is one line. Most were below 0.3.
So we removed the contaminated examples and retrained. We also retrained on the original data with nothing changed but the seed, so we could tell the effect of the data apart from the effect of a rerun.
| Checkpoint | HumanEval | HumanEval+ | MBPP |
|---|---|---|---|
| Base | 13.4% | 11.0% | 20.2% |
| Chat, as published | 26.8% | 25.6% | 38.9% |
| Same data, seed 2 | 26.2% | 25.6% | 34.6% |
| Decontaminated | 22.6% | 21.3% | 35.0% |
Decontaminated versus published on HumanEval: -4.3 points, 8 problems lost and 1 gained, McNemar p = 0.039. The two original-data runs agree to within one problem, so the drop is the data and not the rerun.
Where the lost points came from
If this were memorization, the losses would sit on the 63 problems that matched removed training data. They do not.
| Subset | Problems | Base | Published | Seed 2 | Decontaminated |
|---|---|---|---|---|---|
| In the removed data | 63 | 5 | 14 | 17 | 13 |
| Not in the removed data | 101 | 17 | 30 | 26 | 24 |
Against the mean of the two original-data runs the decontaminated model is about four points lower on both subsets.
The contamination was real and it was not memorization. Practice on HumanEval-style problems helped on HumanEval-style problems generally, including ones that were never in the data. We had reported that as ability.
What MBPP shows, which is a different lesson
Only 10 MBPP problems matched removed data, and the two original-data runs differ by 11 problems. The published 38.9% is best read as a lucky seed. Run-to-run variance on MBPP was about four points, which is the same size as the contamination effect on HumanEval.
A single run reports the model, the data, the harness and the seed as one number.
What to take from this
Scan your instruction data. Docstring prefixes plus solution n-grams. Ours runs in minutes and is in the repo.
Do not trust "decontaminated" as a label. Check what the filter actually matched on. An exact-match filter against a dataset built by rewriting is close to no filter at all.
Run the seed control. Without it we could not have separated a four point data effect from a four point variance effect, and on MBPP the variance was the larger of the two.
Ask vendors three things, including us: the decontamination method, the evaluation harness, and how many runs are behind the number.
Full report with the figures, confidence intervals and McNemar tests: trifero.ai/research/elroy
Code, nine chapters, every benchmark completion and failure record: github.com/strifero/elroy, Apache-2.0.
I run Strife Technologies and the trifero.ai lab. We built this model to be able to answer questions about AI with measurements instead of adjectives, and this is the measurement we liked least.
