Paste this into a file and run it against the JUnit XML your CI already dumps. No install, no API key, no GitHub Actions change. You get a one-line score per test, sorted worst first, computed the same way a deterministic engine computes it.

python flaky_score.py results/*.xml

The reason you should do this before you reach for the model is the one that took me the longest to accept. A flaky test is not a property of a test. It is a property of one test, on one commit, that produced both a pass and a failure. The same green check on Monday and the red one on the rerun of the same build are the evidence. Once that lands, the whole detector collapses to a ratio, and the ratio needs no intelligence at all.

The definition is a pair, not a name

This week a QA engineer published a working detector on dev.to, a small open-source project called flaky-test-detector, built around a scoring engine and an optional AI agent on top of it. I read the write-up in full and worked the scoring logic out of it, because the write-up is where the reusable idea lives and the surrounding scaffolding is where most teams will not want to start.

The definition the engine works from: a test is flaky when, for the same commit, the outcomes of that test include both passing and failing. Skipped results are dropped, because a skip says nothing about reliability. Results that carry no commit identifier are dropped too, because you cannot compare two runs you cannot line up. An error counts the same as a failure. Everything else is noise you have to decide what to do with.

The score per test is then flaky_commits / usable_commits. A test that ran across ten commits and flipped on three of them is at 0.3. The author flags a test as flaky at a threshold of 0.1, and only looks at tests with data from at least five commits, so a score that rests on two runs cannot get a vote. Both numbers are constants you change to match your pipeline; I have not found a reason to move them in normal operation.

I ran the same logic against a fixture I made by hand: three tests, across seven result files. The first test failed on the first run of one commit and passed on the rerun of that same commit, with five other commits behind it where it passed cleanly. The second test passed on every commit. The third test was skipped everywhere. The result is the whole point in one table: the flipper came out at 0.17 with the flag raised, the always-clean test at 0.00 with the flag down, and the skipped test never appeared at all. That is exactly what I want a scorer to do, one line of evidence per test, nothing the model could talk me out of.

The script

Here is the engine as a single file. Standard library only, which is the point: it runs in the CI runner that already produced the XML, and it runs on a laptop that does not have the project installed.

#!/usr/bin/env python3 # flaky_score.py -- usage: python flaky_score.py run1.xml run2.xml ... # A test is FLAKY when the same test, on the same commit, produced both a # pass and a failure. score = flaky_commits / usable_commits. import sys import xml.etree.ElementTree as ET from collections import defaultdict THRESHOLD = 0.1 # flag a test flaky at score >= this MIN_COMMITS = 5 # need this many usable commits to trust a score def outcome(case): if case.find("failure") is not None: return "failed" if case.find("error") is not None: return "error" if case.find("skipped") is not None: return "skipped" return "passed" def commit_of(root): for p in root.iter("property"): if p.get("name") == "commit_sha": return p.get("value") return None def score(files): seen = defaultdict(lambda: defaultdict(set)) for path in files: root = ET.parse(path).getroot() commit = commit_of(root) for case in root.iter("testcase"): name = case.get("classname", "") + "::" + case.get("name", "") out = outcome(case) if out == "skipped" or not commit: continue seen[name][commit].add(out == "passed") rows = [] for name, by_commit in seen.items(): flaky = sum(1 for s in by_commit.values() if len(s) > 1) total = len(by_commit) s = flaky / total if total else 0.0 rows.append((s, flaky, total, name)) rows.sort(reverse=True) return rows if __name__ == "__main__": print(f"{'score':>6} {'flaky?':>6} {'commits':>8} test") for s, f, t, name in score(sys.argv[1:]): flag = (t >= MIN_COMMITS) and s >= THRESHOLD print(f"{s:6.2f} {str(flag):>6} {f:>3}/{t:<3} {name}")

Two lines carry the definition, and they are the lines to read out loud before you trust the output. The seen[name][commit].add(out == "passed") line is where a test on a commit accumulates a set of outcomes; a set that ends up with both True and False is a flaky commit. The len(s) > 1 test is the only place flakiness is decided. Everything else is bookkeeping: keep distinct commits, drop skips, drop missing commits, sort, print.

Where the JUnit has to cooperate

The one real requirement is that each result file names its commit. The engine reads a commit_sha property off the file, which is the shape a per-run JUnit artifact already has when your CI writes one out per build. If your results do not carry a commit id, the engine will happily score nothing, because every line gets dropped as un-comparable. That is the failure mode to check first when the output looks empty: not a bug in the ratio, but the property missing. The original project pulls both values from the GitHub run when it fetches artifacts by repo; the point is the same, a run is only usable if you can say which commit it ran on.

If you are not on GitHub Actions, you are not out. Anything that emits standard JUnit XML with a per-run commit property feeds this. The author is direct that the tool was built with GitHub Actions in mind and may need changes for other pipelines; the score itself is portable, the artifact plumbing is not.

What the model is for, and when to skip it

The detection is done. Nothing below this line needs a model to have happened, and that is the part I want to keep straight, because the easy move is to hand the whole thing to an agent and call it reliability engineering.

The AI step the published project adds is narrow on purpose. An agent reads the JSON report, and for each flaky test it reads the list of distinct failure messages and infers a root-cause category, timeouts, race conditions, network or gateway instability, shared state, and says which tests a cause touches and how confident it is. Then it writes a Markdown note. In the sample report the write-up shows, one of the worst tests carried five different messages, and they map cleanly to two causes, a gateway that times out and a gateway that resets the connection, rather than to five separate bugs. (The sample is the author's illustration of the report shape, not a claim about a specific production run.)

My rule for when to run that pass is budget-shaped, not sentiment-shaped. The score already tells you which tests are flaky and how badly. The model adds the guess about why. If a test is at 0.7 and your error messages already say timeout waiting for gateway, you probably know the cause without spending the call. If a test is at 0.7 and the messages are five shapes with no obvious common thread, that is the test where an extra pair of eyes is worth the tokens. Run the score on every pipeline you care about. Run the agent on the handful of tests where the messages do not already answer the question.

The author's own caveat is the one to carry into your notes. The model gives a first look at the possible causes, and it does not replace a human reading the result, because a failure caused by the infrastructure underneath the test looks like flakiness from the outside. A test that flakes because the runner ran out of memory is flaky in the ratio and not fixable by touching the test. The score will surface it; only the person who owns the environment can tell you it is not the test.

How often to run it

The cadence depends on how often you ship, and the author states both branches plainly. If you deploy continuously, once a week is enough to keep a number on the board and to notice a spike after an infra change. If you ship on a monthly cycle, run it on every deployment and compare the latest run against the last, so the question is always whether the count went up, not whether it is small in absolute terms. Either way, a sudden jump in the flaky count on the day after a runner or shard change is pointing at the environment, not at your tests.

That is the whole thing. A pair-based definition, a ratio, a threshold, a floor on how much evidence a score needs, and a model that you call only on the tests whose messages stop talking. The detector is the part that is boring and portable, and the boring part is the part that is actually doing the work.

This post was written with AI assistance. The author is responsible for its content.