Quick answer: Randomized test data is a good idea; an unseeded random generator is not. Without a fixed seed, every run explores a different input, and when a failure occurs you cannot reproduce it, which makes the bug look intermittent. Use a seeded generator, print the seed on every run, and design the test to fail loudly with the exact seed that produced the problem.
The symptom was a billing test that failed about once every two weeks in CI and never failed on the same engineer's machine twice. It generated a random line-item count and random amounts. One Tuesday the generator produced a quantity of zero, which the discount logic did not handle. Because the seed was never recorded, the rerun used new random values, passed, and the failure was written off as flaky. The bug was real and deterministic for that specific input; the test was just hiding how to get back to it.
Why unseeded randomness breaks trust
There are two distinct uses of random data, and they get conflated. Property-based testing deliberately generates many inputs to find edge cases, but it always records the seed and can shrink and replay a failure. Ad hoc Math.random() sprinkled through a test gives you the exploration without the reproducibility. You inherit the cost of nondeterminism and none of the machinery that makes it manageable.
Here is the problematic pattern and why it fails.
When quantity happened to be zero and the amount was nonzero, the discount calculation divided by total quantity and produced a value the assertion did not expect. Nothing in the output told anyone which input caused it.
What a reproducible setup looks like
A seeded generator makes the sequence deterministic and replayable. The table below shows the properties I require before trusting randomized inputs.
| Property | Unseeded ad hoc random | Seeded generator |
|---|---|---|
| Reproducibility | Impossible to replay a failure | Same seed reproduces the exact input |
| Failure diagnosis | Looks like flakiness | Seed points straight at the edge case |
| Coverage | Random, uneven across runs | Can vary seed per run while staying replayable |
| Debugging | Rerun changes the input | Replay the failing case in isolation |
Seeding the generator and surfacing the seed
Use a deterministic pseudo-random generator and derive the seed from an environment variable or the current time, but always print it. That way a normal run still varies, while a failed run can be replayed exactly.
To replay the Tuesday failure, the next run is a single environment variable: TEST_SEED=<printed value>. The generator produces the identical sequence, including the zero quantity, and the bug now fails on demand.
Generate valid inputs, and test the invalid ones explicitly
A generator should usually produce values within the documented domain so the test is exercising real behavior rather than tripping on inputs the API rejects. Define the bounds deliberately, including the edges, rather than multiplying an unbounded random number.
That does not mean zero is ignored. Once the seeded test exposed the unhandled zero, the correct response was a dedicated, explicit test for a zero quantity and a fix in the discount logic, not a hope that the generator avoids it. Random exploration finds the edge case; named, fixed tests lock the behavior so it never regresses.
Property-based frameworks do the hard parts
When you want genuine input exploration, reach for a property-based library rather than hand-rolling. It manages the seed, runs many cases, and shrinks a failing input to the smallest value that still triggers the bug. A failure that originally needs a ten-item list with a zero in the middle can be reduced to the single zero that actually matters. That shrinking step is the difference between a confusing failure and a one-line repro.
FAQ
Is random test data a bad practice?
No. Exploring varied inputs catches edge cases that hand-written fixtures miss. The bad practice is unseeded randomness that cannot be replayed. Pair random data with a fixed, logged seed and it becomes a strength.
How do I reproduce a failure that happened in CI?
Read the seed printed in the failed run's log and rerun the test with that seed. The generator then produces the identical sequence. If the seed is not logged, add that before anything else; without it, the failure is effectively unreproducible.
Should I just pin one seed forever?
A pinned seed gives reproducibility but no new coverage over time. Prefer a varying seed per run that is always recorded, with the option to override it for replay. Keep a few fixed seeds or explicit edge-case tests as permanent guards.
Why did the bug look flaky instead of real?
Each rerun generated different inputs, so the failing condition appeared only occasionally and could not be triggered again. The underlying defect was deterministic for its input; the nondeterministic test was masking it. Recording the seed removes the illusion of flakiness.
This is part of the API Testing Guide — start there for the patterns that keep tests deterministic, isolated, and trustworthy.
