My AI shipped a 1,199-line change nobody understood. I built a CLI that scores how fragile it is.

An essay made the rounds this week: "AI usage generates a fragile codebase." The argument is simple. AI writes code faster than humans can review it, so the bottleneck moved from writing to the review gate. Code that looks right, passes the easy tests, and ships before anyone understands it.

I have felt that exact failure mode. A 1,199-line change lands, the tests are green, the diff is too big to read properly, and two weeks later nobody on the team can explain what half of it does. The review happened. Understanding did not.

So I built fragility: a zero-dependency TypeScript CLI that measures four fragility modes of AI-written code on the diff you are about to merge, and returns a CI verdict.

GitHub: https://github.com/techdevfun/fragility

What it does

Three commands, no API key, no server, no model calls:

fragility audit --base HEAD~1 fragility audit --diff pr.diff --format json fragility explain review

audit scores four signals, each 0-100, plus a composite FRAGILITY SCORE:

  • Test shallowness - changed lines with no test asserting them. Orphan lines are the core number: lines no test covers.
  • Over-abstraction - call depth, parameter counts, and branch counts on changed functions. Deep chains and wide signatures are harder to hold in your head.
  • Comprehension debt - how much of the change comes from commits nobody on the team has touched since, and how fast lines are being merged. AI-marked commits are matched against the full commit message, including Co-Authored-By trailers.
  • Review bottleneck - diff size mapped to a detection curve, plus lines merged per day against a 400 lines/day heuristic default.

The verdict is ROBUST, WATCH, or FRAGILE, with exit codes made for CI: 0 robust, 1 fragile, 2 bad input. Everything runs 100% locally. The tool never calls an API, never judges code with a model, never sends anything anywhere.

I ran it on a real-shaped change -- and the numbers are stark

fragility demo

On a synthetic seeded fixture of a 1,199-line AI-style change, the audit reports FRAGILE 20/100, exit 1: 100% orphan lines (1,169 of 1,169), review velocity 1,225 lines per day. The same audit on a small healthy change reports ROBUST 96/100, exit 0.

On a single fragile diff file, the signals read 10/45/85, composite 47/100, exit 1. Add an LCOV coverage file and the composite moves to 43 -- the coverage data confirms what the assertion scan already suspected. A healthy diff scores 100/100/100.

The most useful number is the orphan-line count. It is checkable: git blame shows exactly which lines no test touches. That is the whole point of the tool. It does not give you an opinion about your code. It gives you a number you can verify with a command you already know.

The bug CI caught that my tests missed

The first version of the tool read only commit subject lines when looking for AI markers. That sounds fine until you remember where Co-Authored-By trailers live: in the commit body, never the subject. So a real AI-marked commit, marked the standard way, scored 0% orphan lines and passed the gate with a WATCH verdict -- a fragile change sailing through the exact check that was supposed to stop it.

Every test in the suite had put the marker in the subject line, so the suite was green while the feature was broken for the real-world convention. The fix reads the full commit message, and the test suite now includes the realistic case: a body-trailer commit with a deep call chain and one weak test. That scenario now scores 59/100 FRAGILE, exit 1 -- exactly what the first review predicted it should.

Where the rivals beat us

  • patchcov / Target PR coverage measure one signal: which changed lines tests cover. They do it well. fragility adds three more signals in the same command, but if all you need is coverage on a PR, use them.
  • gapix checks assertions at the AST level, which is deeper than my regex-based assertion counting. My counter under-counts table-driven and dynamic tests, and the README says so. gapix does not scope its check to the diff or combine it with review load.
  • CodeRabbit and Qodo are full LLM reviewers in the cloud. They read the code and write comments a human would write. That is stronger than any deterministic signal -- and it costs money, sends your code to a third party, and needs an account. fragility is offline, free, and deterministic.
  • LinearB is a SaaS dashboard for engineering metrics. It will show you review velocity across the team. It will not gate a single PR in CI with one command.

fragility does not replace any of these. It is the cheap pre-merge check: four numbers, one verdict, one exit code, no network.

Honest limits

  • Assertion counting is regex-based. It under-counts table-driven and dynamic tests. The tool reports this as a stated limitation, and LCOV input sharpens the number when you have it.
  • AI-marker detection is a heuristic, never authorship proof. A commit is flagged because its message says an AI wrote it, not because the tool knows who typed it.
  • The detection curve is population data from published PR studies, reported as an estimate. The 400 lines/day review capacity is a configurable heuristic default, not a measured constant.
  • The tool measures the diff you give it. It cannot tell you whether the tests themselves are good -- only whether they assert on the lines that changed.

Try it 

git clone https://github.com/techdevfun/fragility.git cd fragility npm install npm run build node dist/src/cli.js audit --base HEAD~1

Install is from source for now. npm publish is pending. The repo ships the seeded fixtures and the demo transcript, so you can reproduce every number in this post yourself. If your AI writes faster than your team can understand, this is the cheapest way I know to measure the gap -- and to prove it with a file, not a feeling.