The complaint arrived as a screenshot: a spinner, then an error page, on a Tuesday afternoon. Three customers could not pay. By the time I opened the dashboard for that service, everything on it was green. Four instances up, CPU under 20%, uptime for the month a shade over 99.9%, no alert fired. The /health endpoint had answered 200 the whole time.

Here is that endpoint, with the variable names changed:

// routes/health.js router.get('/health', (req, res) => { res.status(200).json({ status: 'ok', version: pkg.version, uptime: process.uptime(), }) }) 

That endpoint cannot fail. It reads nothing and calls nothing, so it keeps answering 200 while Postgres refuses connections, which is exactly what it did for about 40 minutes. The service is around nine years old, roughly 40k lines of Node and Express, 214 specs in the suite, and a coverage badge in the README that reads 87%.

The people who own it asked a fair question. The suite is green, coverage is high, so why did nothing catch this? Coverage answers how much code ran while the tests ran, not which failures anything in that repo can notice. I wanted the second answer before touching a single spec, and there is a cheap way to get it.

Borrowing determinism from a much bigger idea

Martin Fowler's site published an article by Unmesh Joshi on deterministic simulation testing. The technique replaces the real clock, network and storage with simulated components driven by a single threaded execution loop, so the same partial failure produces the same execution trace on every run. It is written for distributed data stores, using a quorum key value store as the worked example. A nine year old checkout is not a distributed data store, and nobody there is going to rebuild it around a simulation loop this quarter.

The part I could copy in an afternoon was the determinism. One seed, one injected fault, the same trace every time. I had already tried the other way: random packet drops and random connection kills against a staging copy. Two runs produced two different traces, the team called it flaky before lunch, and the conversation died there. A fault that fires on query number 3 on every run is an argument instead of an anecdote.

The three things I instrument before trusting the suite

1. A health check that executes the dependencies

First the endpoint has to do work, with a timeout short enough that it never becomes the outage:

router.get('/health', async (req, res) => { const checks = {} try { await pool.query('SET statement_timeout = 800; SELECT 1') checks.db = 'ok' } catch (err) { checks.db = err.code || 'fail' } try { await gateway.ping({ timeoutMs: 800 }) checks.gateway = 'ok' } catch (err) { checks.gateway = 'fail' } const ok = Object.values(checks).every((v) => v === 'ok') res.status(ok ? 200 : 503).json({ checks }) }) 

Two dependencies, two codes in the body. The load balancer now has something to act on, and so does the on call person.

2. A seeded fault injector around the pool

No new library, no sidecar. A wrapper with a counter, so the failure lands on the same query every run:

// test/faults.js function faultyPool(pool, { failOnQuery, latencyMs = 0 }) { let n = 0 return { async query(sql, params) { n += 1 if (n === failOnQuery) { const err = new Error('connection terminated unexpectedly') err.code = '57P01' throw err } if (latencyMs) { await new Promise((r) => setTimeout(r, latencyMs)) } return pool.query(sql, params) }, } } 

The counter is the whole trick. failOnQuery: 3 kills the third query of the request, which in this codebase is the one that reserves stock. Same seed, same victim, every single time.

3. One counter for the business event

Uptime is not a thing a customer buys. Failed payment attempts are:

paymentAttempts.inc({ result: 'fail', reason: err.code || 'unknown' }) 

One line in the catch block of the checkout handler, so the answer to "did anyone fail to pay yesterday" is a number somebody can read on a Monday morning.

The run

I wired faultyPool into the test bootstrap, with failOnQuery: 3 and 900 ms of latency on every other query, and ran the whole suite.

214 specs. 3 turned red, all three in a module that formats order emails. 61 specs exercise the payment flow, and 58 of them replace the pool with a stub that always resolves, so the injected fault never reached the code under test. Nine of those 58 call the handler and assert nothing past the HTTP status.

Then I ran the same fault against the service itself, by hand, with the gateway in sandbox mode. The response to the browser was HTTP 200 with { "status": "pending" }. The front end renders that as "we are confirming your payment". No payment row was written, no order existed, and the error from Postgres died inside a catch (err) { logger.warn(err) } three frames down. The dashboard stayed green because nothing in the request path had any opinion about failure.

What did not work

My first instinct was mutation testing, since the coverage number was the thing nobody trusted. I pointed Stryker at the repo and left it running. A bit over 3 hours later it reported surviving mutants, mostly in a date formatter that two call sites use. Useful information, wrong order. Running it later against the payment module alone took around 18 minutes and told me something I could act on.

The random chaos attempt I already mentioned. It cost me two days and a reputation hit.

Where it stands

The drill is a nightly job now, with three seeds: fail on the first query, fail on the query that writes the payment row, and 900 ms of latency on everything. The swallowing catch is gone, the handler returns 5xx, and the alert fires on the failed attempts counter rather than on CPU. In the first week that counter showed 11 failed attempts on a day the old dashboard called perfect.

The thing I have no good answer for is the 58 specs that stub the pool. Rewriting them as integration tests is weeks of work on a service nobody wants to touch. Keeping them means the suite will stay green through the next outage of this exact shape.

If you have worked on something like this: do you keep those stubbed specs and add a thin layer of fault tests around them, or do you delete them and accept the coverage drop in front of management? And how do you inject a deterministic failure into a dependency when you did not write the service and there is no staging that looks like production?