Last week I deleted data from my own product. Not a test fixture, not a staging artifact — live data in the readiness platform I am building. The dashboard showed webhook heartbeats that had never fired, a lab node presented as if it were a monitored system, and "verified control" counts that no verification had ever produced. None of it was there to deceive anyone on purpose. Each piece had arrived the same way: I needed a screen to look right while I built the thing behind it, so I put something plausible in its place, and then I kept building and forgot to come back. The screen looked complete. The product was lying.
I deleted all of it. Then I rebuilt the ingestion so that a number only appears if a real event produced it. This is the story of what that rebuild taught me — not about a framework or a cloud service, but about the engineering choices that decide whether a security score means anything at all.
How the fake data got there
Every solo builder knows this pattern. You are building a dashboard, and the dashboard needs data before the pipeline that produces the data exists. So you seed it. A heartbeat timestamp here, a control count there — just enough to see the layout work. It is the right call in the moment. The mistake is not seeding it. The mistake is shipping the seed as if it were evidence.
In my case the seeds were of three kinds. The first was synthetic telemetry: webhook heartbeats that fired on a timer inside the app rather than arriving from a real integration, so the dashboard showed a healthy stream that no external system had ever sent. The second was a lab node — a machine I run for testing — listed alongside real monitored systems with no label distinguishing it. The third was the worst: a count of "verified controls" derived from configuration I had written by hand, not from any check that had run. The number looked precise. Precision is what made it dangerous. A vague claim invites scepticism; a specific number invites trust.
The uncomfortable part is that I only found all three because I went looking. Nothing had broken. No customer had complained, because there was no customer yet — but the habit was already formed, and habits formed before the first customer are the ones that survive contact with the first customer. That is why I deleted the data instead of labelling it and moving on. A label would have been honest enough for a screenshot. It would not have fixed the pipeline that let fake evidence into the system in the first place.
Lesson 1: a score is a claim about evidence, not a number
The scoring engine in my platform is deterministic: the same evidence always produces the same score, because the score is computed by plain Python rules and never by a model call. The model writes the narrative around the score — the explanation in plain language — and nothing else. I built it that way on purpose, because a score that changes when you rephrase the prompt is not a measurement, it is a mood.
But determinism only protects you if the inputs are real. A deterministic function over fabricated evidence produces a perfectly reproducible lie. That is the lesson the fake heartbeats taught me: I had spent my care on making the scoring rules correct and almost none on making sure the evidence feeding them was genuine. The fix was to move the honesty boundary upstream, to the ingestion layer, so that nothing enters the scoring pipeline unless it carries proof it happened.
The practical version of that is simple and boring, which is why it works. Every ingestion event now arrives with an HMAC signature over its payload, verified against a secret held per source. An event without a valid signature is rejected, not stored. Credentials for those integrations are encrypted at rest with AES-256-GCM. And every event carries its source and its received time, so the score can say not just what it is but when the evidence behind it was last fresh. Boring is the point. The exciting parts of a security product should be the findings, not the plumbing that decides whether the findings are real.
Lesson 2: "unable to verify" is a feature, not a failure
The hardest decision in the rebuild was what to show when there is no evidence. The old dashboard never had that problem, because it always had something to show — that was the problem. The new one has four states: ready today, action needed, critical risk, and unable to verify. That last one is the most important, and it was the hardest to keep.
Every instinct in product building pushes against a screen that says "I don't know." It feels like an unfinished feature. But for a readiness score, "unable to verify" is the only honest answer when the evidence is missing, and it is also the most useful one: it tells the operator exactly what to fix. A control that has never been checked is not a passing control. It is an unknown control, and unknowns are what incident response is made of. I have spent enough time in DFIR work to know that the gap between "we thought it was fine" and "we had never actually checked" is where breaches live.
So the rule now is: no evidence, no score. Not a low score, not a neutral score — no score, and a clear statement of what is missing and what would produce it. That rule cost me a dashboard that always looked complete. It bought me a product I can defend.
Lesson 3: version the rules, or you cannot explain a score from last month
The third change came out of a question I asked myself while rebuilding: if a customer challenges a score from three weeks ago, can I reproduce it? With deterministic rules the answer should be yes — same evidence, same rules, same score. But that is only true if the rules have not changed since, and rules always change. A threshold gets tuned, a control gets redefined, a check gets added. If the score does not record which version of the rules produced it, then last month's score is unreproducible, and an unreproducible score is an undefendable one.
Every score now carries the rule-set version that produced it, and the rule set itself is versioned in the repository like code, because it is code. Old versions stay available so any historical score can be recomputed from its stored evidence. This is a small amount of bookkeeping and it changes the character of the product: a score stops being a snapshot and becomes an auditable claim. In incident readiness, auditability is not a compliance checkbox. It is the difference between a score you can act on and a score you have to take on faith.
Lesson 4: the lab stays in the lab
The lab node taught me the simplest lesson of the four, and the easiest to forget. A test system that appears in the same view as production systems, with no label, is a false positive waiting to happen — or worse, a false negative, because a real system that goes quiet looks normal next to a lab node that was never supposed to be loud. Test data and production data now live in separate projects with separate credentials, and nothing from the lab can reach the scoring pipeline at all. The separation is enforced by infrastructure, not by discipline, because discipline is what failed the first time.
What I kept
I did not throw everything out. The deterministic scoring engine survived, because the design was sound — the failure was in what fed it. The four-state readiness report survived, because the states were the right abstraction; they just needed to be fed honestly. And the habit of building in public survived, because writing about the fake data is what made the rebuild real. If I had fixed it quietly, I would have fixed the code and kept the habit that produced it. Saying it out loud is what makes the standard stick.
The platform now shows fewer impressive numbers than it did a week ago. Some screens show "unable to verify" where they used to show a confident count. That is the trade, and I would make it again: a readiness score is only useful if it is true, and truth in a security product is not a marketing claim. It is an engineering property, built out of signed ingestion, encrypted credentials, real health checks, honest sync states, and versioned rules. Everything else is a dashboard that looks finished while the product underneath it is still pretending.
