The one word an on-call engineer reads first in an AI root-cause report is the confidence. At 3 am, "HIGH" means "act on this". So the most important question about any AI SRE tool is not how often it is right. It is how often it says HIGH and is wrong.
I build one, AlertiGate, a self-hosted tool that investigates Kubernetes alerts. When I finally measured that number, it was bad. This is what I found, what didn't fix it, and what did.
How the investigation works
The details matter for what went wrong, so briefly:
- An alert fires. The model proposes a few hypotheses ("the new release cut the memory limit", "the node is under pressure").
- For each hypothesis it writes predictions: things that must be true if the hypothesis is. "If the limit was cut, the pod spec's limit differs between revisions."
- It runs real queries against Prometheus, Loki and the Kubernetes API to test them.
- A second model call, the judge, reads each prediction beside the query results and answers CONFIRMED, DISCONFIRMED, INCONCLUSIVE or UNTESTED.
- Confidence is computed in code from the evidence: how many predictions were confirmed, how direct and complete the evidence is, and how clearly the hypothesis beats its rivals.
The model never states its own confidence. "I am 90% sure" from an LLM is a sampled token, not a probability, so the number comes from a pure function with test vectors that fail the build if a weight change moves a band.
I thought that made HIGH safe. It didn't.
Measuring it
I seed faults into a lab k3s cluster where I know the real cause: an OOM after a release lowered a memory limit, a broken liveness probe, a missing Git repository behind an Argo CD app, and so on. I also run fixtures in HolmesGPT's open test format. Each investigation costs a few cents.
Then the tedious part: every hypothesis in 57 runs was labelled true, co-true or false against the known cause, one by one, under a written labelling rule.
Result: 9 false hypotheses reached HIGH, in 7 of the 57 runs. Roughly one run in eight would have told an on-call engineer, with the strongest label available, to fix the wrong thing.
Why HIGH was wrong
Tracing every false HIGH back to the records that carried it gave three causes. None of them was "the model hallucinated".
1. Facts that are true whatever the cause. "The pod restarted." "The release differs from revision 1." Those are true in almost every incident after a deploy. A false hypothesis that predicts them gets them confirmed, and its coverage looks excellent.
2. Shared symptoms. OOMKilled supports "the limit was cut", "a memory leak" and "a traffic spike" equally. Evidence that supports every rival should count for none of them, and at first it counted for all of them.
3. The judge confirming half a prediction. 7 of the 56 confirmations given to false hypotheses came from two-part predictions where only one part was shown: "RELEASE differs between revisions and points to a larger index", confirmed from a record that showed only the first half.
What didn't work
Telling the judge in the prompt. I added a clear rule: "confirm a prediction of several parts only if every part is shown". Replayed on the same inputs, the judge kept confirming 5 of the 6 cases, 3 times out of 3 each, sometimes writing in its own rationale that a part was not shown and confirming anyway.
Asking the model to mark its key prediction. The idea: have the proposer mark the one prediction that tests its hypothesis's actual reason, and only allow HIGH once that one is confirmed. Re-scoring old runs offline, false HIGHs dropped from 9 to 1. It looked like the fix.
Live, it made things worse. The proposer marked a fact true whatever the cause as its "reason" for 11 of 12 false hypotheses, so the gate never closed on them. False HIGHs rose to 3 runs in 12, and some true causes got worse.
That was the most useful lesson of the whole exercise: an offline re-score that assumes the model will follow a new instruction is not a measurement. I had measured my assumption, not the model. I removed the gate.
What worked
Structure the judge's answer, and enforce it in code
Instead of a rule in prose, the judge now has to list the prediction's parts, each marked shown or not shown. The parser refuses CONFIRMED if any part is marked not shown. The model can still be wrong about a part, but it can no longer write "not shown" and confirm in the same breath.
Overreaches on the replay set went from 21 of 24 to 3 of 24, and every control (true two-part predictions, fully shown) stayed confirmed.
Challenge every HIGH before showing it
The rules that score evidence decide which hypotheses could be HIGH. Each of those is now challenged: the judge lists the hypothesis's own claims, the actual cause it asserts, and checks each against all of the investigation's records. HIGH stands only if every claim is shown. Otherwise the hypothesis is capped at MEDIUM, and the report says which claim wasn't shown.
This attacks cause 1 directly: a hypothesis that got its coverage from facts true anyway still has to show its own claim. Replayed, the 8 false hypotheses that had reached HIGH failed the challenge 24 times out of 24.
It's a cap, not a refutation: the challenge can only lower a band, never call a hypothesis false. Of those 24 failures, only one said a claim was contradicted; the rest said it simply wasn't shown. That is good evidence for "not HIGH", and weak evidence for "false".
Fix the data before blaming the model
The challenge also failed some true causes, and one looked like a model mistake. The true cause claimed the process needs more than its 48 Mi limit, and the judge kept finding that not shown. Beside it sat a record saying memory was "flat at 4.26 MiB for five minutes".
It was a data bug. The investigation's time window ended 30 minutes after the alert, so range queries asked Prometheus about the future. Prometheus fills each step with the latest sample within its 5-minute lookback, so one real sample became twenty points of a flat line. 18 of 62 metric records in one suite were stretched like that.
The fix was in the code that turns query results into evidence: points after the moment of the query fold back into the one real sample, and a range wholly in the future can no longer be read as "nothing there". After it, the true causes passed the challenge 21 of 21.
A related rule came from the same kind of digging: a reading only counts where it could have seen. cAdvisor scrapes every 10 seconds; a container that is OOM-killed within a second of loading its index never shows its peak in any scrape. "Peak memory was only 4.3 MiB" refutes nothing there. The termination itself is the observation.
Don't let a false rival hold back a true cause
The last fix ran the other way. Often the true cause sat at MEDIUM because a rival wasn't "clearly weaker". In every such case I checked, that rival was false. Now a hypothesis held back only by its rivals is challenged, and a rival that fails its own challenge is set aside. Replayed, those rivals failed 17 of 17 times and the true causes passed 15 of 15.
Results
Zero false HIGHs in the 29 measured runs after the challenge shipped, and none in the larger measurement runs since.
The whole calibration effort, every paid run and replay, cost under $2 in model calls. The expensive part was the labelling.
What it cost, and what's still open
HIGH is now rarer than it should be. When the true cause's evidence is shared with its rivals (OOMKilled again), none of its predictions tell it apart, so it stays at MEDIUM with thin coverage. My hardest scenario passes in only 1 or 2 runs out of 5, mostly for this reason.
I chose that trade on purpose: a true cause at MEDIUM costs the engineer a minute of checking; a false cause at HIGH costs them the incident.
If you build or buy one of these
- Measure false HIGHs, not accuracy. Label every hypothesis against a known cause. "It found the root cause 90% of the time" says nothing about what it told you the other 10%.
- Never let the model state its own confidence. Compute it from evidence you can show.
- Put rules in structure and code, not in the prompt. The judge broke a plainly written rule while describing the violation.
- Replay the step you changed. If a fix depends on the model doing something new, only a live run measures it.
- Check the data the model sees. My worst "model error" was a query window ending in the future.
- Prefer a cap to a verdict. A check that can only lower confidence is safe to add; one that can call things false needs far more proof.
I'd especially like to hear from anyone who has measured confidence in their own tool: what did you count as a false HIGH, and what moved it?
Disclosure: I'm AlertiGate's founder. It was built with heavy help from AI coding tools, which also did much of the measurement and labelling described here under my direction. This post was drafted with AI from the project's measurement notes and the code's own documentation.

