When a proptest test passes, cargo test prints ok, and I kept wondering whether 256 test cases had been enough. So I wrote proptest-residual-risk, which prints three more lines for every passing test:
- an upper bound, at 95% confidence, on the chance that the next test case fails;
- how many regions of the code under test the test cases ran;
- an estimate of the chance that the next test case runs a region none has run.
To try it, add it as a dev-dependency and import its proptest! in place of proptest's; the tests stay as they are. The coverage numbers need the coverage feature and cargo llvm-cov.
It only sees tests written with proptest!, and the numbers are about inputs from the test's own generator, not inputs in production. With coverage on, a test takes about 2.25 times as long as with plain proptest (about 1.06 times without it, geometric mean over 11 properties from real crates). How each number is defined is in docs/theory.md.
I'd like to hear whether the numbers make sense on your own tests, and where they don't.
Written with the help of an AI assistant; I have read it and take responsibility for it.
1 post - 1 participant
