Whose Bug Was It

Twenty-three defects in the last twenty-two runs: eleven in the collector, four in config, three in other services, and five of our own.

👋 Hi, I'm Anton - a software engineer. These days I work on a platform: contracts, generated code, data paths and the tooling around them. Running notes live on my GitHub: github.com/brilliant-almazov.

The count covers only the last twenty-two acceptance runs of the week, where every fix is itemised; one collector fix is still in review and the three defects in other services are open and planned, and both are marked that way below.

This is part 4 of the series on one week of integration acceptance across a whole chain of services, after months of doing it the slow way. Maybe you already do it better, maybe you'd do it another way - this is how it went, and where it stops.

In real life

The whole system is a zoo. The services are the residents, the bus is the paths, and acceptance is the vet - Pickle, a guinea pig with a card for each of eighteen enclosures. Nora is the keeper.

Pickle's week of rounds found plenty. Biscuit, the courier dog, had a sore paw, and when Pickle confirmed it he said "Told you so!" with great satisfaction. Miso the cat was wearing a name sign that said BISCUIT, and observed, without opening her eyes, "That sign is not mine." Nora went to fix the sign. At one enclosure the cure was next door - and Pip, the guide, pointed out that next door was closed.

And then, five times in one week, the problem was Pickle. Mixed-up cards, an old page in the notebook, a medicine bottle already empty when she tried it again. Pickle admitted it: "I mixed up the cards." Ember the dragon, who misses nothing, added: "Five times. I counted."

The sore paw is a defect in the collector's code. The wrong name sign is configuration on the stand. The closed door next door is a defect in another service, which the acceptance agent can find and cannot fix. And the vet's own mistakes are the acceptance side's false reds. The honest answer to "whose bug was it" has four parts, and one of them points back at the tester.

The count

The acceptance agent counted defects only over the last twenty-two runs of the week, because those are the runs where every fix is written down item by item. Earlier runs found things too; they are not in this number.

Eleven in the collector

The collector's eleven are the ordinary findings of driving a chain end to end: a status probe that could time out, a watchdog whose threshold did not arrive where it was needed and whose probe read the order's scenario, a queue for rebuilding a buffer, the body of a task post, a request-log row that was not closed when the answer never came, a delivery that had to happen exactly once, a priority that had to be clamped, configuration defaults, per-call balance answers in the emulator, and a number type in one provider's override.

Two of them are worth telling. Priorities outside 0..100 were refused with an error. The owner's norm is different: a number on a scale is clamped to its bound, no error. After the fix, six rows went from red to green. And configuration defaults moved to one form for every leaf, so that a binary starts with sane values when a key is not set:

# every configuration leaf, the form (paraphrased) WATCHDOG_STATUS_TIMEOUT=${WATCHDOG_STATUS_TIMEOUT:-10s} <KEY>=${<KEY>:-<default>} 

The one still in review is the most instructive. The emulator lets a test override its answer per call, and the balance override worked for one provider - five calls of five - and not for the other, zero of five, because a JSON number fell back to the fixture. The lesson the acceptance agent wrote down: measure each provider separately; a pass on one does not prove the other. The fix is written and waits for its rerun.

# an emulator scenario override, answer by call number (form, values elided) {"calls": {"N": {...}}} one provider 5 of 5 calls took the override the other 0 of 5 calls took the override | fix in review, rerun pending 

Four in configuration and the stand

These were not code. The account list for the balance poll needed a fix. An alert called "emulation in production" never fired, because its job label did not match - fixed on the stand. A content service was raised switched off; the switch was named and turned on. And a binary carried another binary's configuration keys: thirteen binaries were checked, none was clean, and after the cleanup all thirteen were. The rule that came out of it is short: a binary knows only its own keys.

Three in other services - open

Three defects sit in services the acceptance agent does not own, and the rows they break stay red until those services change:

• results are delivered to a queue that has no consumer, so they stay unread;
• a status call with tenant 0 is refused with "tenant_id must be > 0", so the empty-result mark is lost;
• two norms for the collector contradict each other, and the choice between them is the owner's.

All three are planned in their services. They are the reason the plan stands at 749 of 753 and not at a hundred, and the previous part of this series explains why they stay red rather than leave the scope.

Five of our own

This is the part I would most like other teams to count. Five defects were the acceptance side's own:

The stale build was the costliest. Verdicts were measured on a build that did not contain the fix under test, and one run showed the plan falling from 732 rows to 728 - four rows lost that had never been looked at on the right code. The fix was a per-build ledger on the acceptance side and a rule: a red mark counts only on the build that carries the fix.

The spent scenario was the most embarrassing. A retry reused an emulator scenario whose call counter was already used up, so the second attempt failed for a reason that had nothing to do with the collector. Now every attempt gets a fresh copy of the scenario, and a test keeps it that way.

The three wrong citations were found from the other direction: when the collector turned plan rows into CI tests named after them, three rows turned out to quote the wrong source. They were fixed in the plan.

Eleven fixed, four added

In the same window the chain did not only get repaired; it also got new behaviour, each piece with its own plan rows:

• the acceptance control calls moved onto a bus;
• four new calls - a balance poll, page outcomes, balance changes and a utility poll - arrived with new plan rows;
• alert rules are now generated into config;
• the scenario id and the emulation flag are written to the request log.

The sign on the door

What went wrong

Five of twenty-three is about one in five. The false reds among them - the spent scenario and the stale build - made the collector look worse than it was, and the wrong citations and key names made the plan itself less trustworthy until they were found. The most expensive was the simplest: a verdict without the name of the build it was measured on.

What I do not claim

• Not that twenty-three is all the defects of the week. It is the count over the last twenty-two runs, where every fix is itemised.
• Not that the in-review fix works. It waits for its rerun.
• Not when the three open defects close. They are planned in services the acceptance agent does not own.
• Not that the acceptance side is now free of its own mistakes. It has fewer ways to make the same five.

What of this is method

• Count your own. A tester's false red is a defect, and it goes into the same table.
• One provider is not proof for another. Measure each one separately.
• Open is a status, not a hiding place. A defect in someone else's service stays red and named.

Eighteen domains in a week - Part 4. This closes the series.

If your test reports count defects by service, is there a row for the tests themselves? If you test against more than one provider, do you measure each separately? If a fix belongs to another team, does the failing row stay red, or does it leave the plan? Whose bug was your last one?