I ran an AI diff reviewer on 79 real pull requests: 4 bugs maintainers fixed, all in the first 29, none in the next 50

By Spacexcx · November 2026

I built a small tool that reads a git diff and flags possible logic bugs, quoting the exact line behind each flag. Between late August and late October 2026 I ran it on 79 public pull requests and checked every flag by hand.

The result is lopsided. Four flags were accepted and fixed by the authors, and all four came from 14 pull requests I picked by hand. In the next 50, which I picked with a mechanical filter, nothing survived review. This post covers what I measured, what I suspect explains the gap, and what I can't tell apart.

Disclosure: the tool is AI-based (Gemini Flash, free tier). I read every flag against the code myself before reporting anything. I wrote, verified, and sent every comment and email myself; the tool only analyzed diffs locally on my machine and never posted anything to GitHub.

TL;DR

• 79 PRs counted, 9 flags in total.
• 4 flags were fixed by maintainers, 4 I dismissed after reading the code, 1 never got a reply.
• All 4 fixes came from 14 hand-picked open PRs. The 50 filtered PRs produced 0.
• On a small blind test of 5 known past regressions it caught 4 (n=5, so treat that as an anecdote).
• Nobody has installed it yet. I tried one community post: 0 replies, 0 installs.

What the tool does

It sends a diff to a Gemini Flash model, then runs a second pass that critiques the first pass's flags before I see them. Every flag has to quote an exact added or removed line, and the code checks that quote programmatically. It ignores Markdown and documentation files, lockfiles and unchanged context lines. It sees the diff only, not the rest of the repository.

How I ran it

Left out of the counts: two PRs from projects above my star ceiling (curl and sqlmodel, one dismissed flag between them) and three Transmission PRs that exceeded the 8-file limit and were never scanned.

What counted as "fixed"

A flag counts as fixed only if the author acknowledged it or changed the code. I'm reporting what the authors did. I am not claiming that each one was a proven crash or exploitable.

Results

The four fixes

1. driftbench #15 (fixed in PR #26): a zero-length insert opcode case in difflib-based code. The author fixed it and merged it with a reporting credit.
2. Godot #119097: a possible null dereference in Joint3D. The author thanked me and fixed it.
3. Godot #123641: a recursion problem in node descendant path handling. The review thread was resolved and the code updated.
4. PyTorch #197776: an import-time crash in the dynamo memory variables code, fixed by the author in a commit (2b2c4fc) that touches only memory.py.

The flags that were wrong or unresolved

I dismissed four flags after reading the code: a deadlock claim that wrongly treated a recursive mutex as non-recursive (the critique pass caught that one), a concurrency warning, a boolean-complexity warning, and a control-flow warning about a loop that was refactored into std::all_of and only behaved differently on already-corrupt input. One more flag, about a possibly undefined name in an open firmware tool PR, never got a reply, so I don't know whether it was right.

The full raw dataset, with PR numbers and zero author names, is documented in the repository: https://github.com/Spacexcx/pr-check

The blind test

I ran the tool on 5 known past regressions (2 Python, 3 C). It flagged 4 and missed 1 on first pass, a host-matching regex issue in a web library. With 5 cases the uncertainty is huge, and the curl cases are well documented, so the model may have seen them. I wouldn't read "80% recall" into it.

The gap

On the 14 hand-picked PRs, 4 produced a fix (29%, with a rough 95% interval of about 12 to 55%). On the 50 filtered PRs, 0 did. If the true rate under that filter were 6% or higher, seeing 0 in 50 would be unlikely, so I'd guess it's lower. If the four fixes had landed at random across all 64 independent PRs, the chance of all four falling in the 14-PR group is about 0.002. So the gap is unlikely to be noise. That doesn't tell me why it exists. These are my candidate explanations, and I tested none of them:

1. My targeting. In the first set I measured the tool plus my instinct for where bugs hide.
2. Repository complexity. Godot and PyTorch are huge codebases with a lot of shared state.
3. The filter. Small PRs, 1 to 8 files, from authors with a public email, in repos with strong CI. Every PR in the 50-PR batch was open and actively under review when scanned, not already merged.
4. The tool changed. The evidence check and the critique pass were on from the hand-picked phase onward, but the model and the prompt changed between phases, and the C/C++ run used a lighter model. I can't rule this out.

One more data point cuts the other way. The 15 tuning PRs were also Godot, and they produced no fixes. They were already merged and I had tuned on them, so I can't read much into it, but it suggests that repo size alone doesn't explain the gap.

What I'd do differently

Pick PRs at random from the same repositories, freeze the model and the prompt before starting, decide the selection rules in advance, and record whether each PR was open or merged.

One more result

I posted about the tool once, in a developer community showcase. It got no replies and no installs in five days, and the repository has 3 stars. If you want to try it for a week on a side project or an open-source repo, I'd like to hear what annoyed you and why you kept it or removed it. Removing it is a totally useful answer. Your diff goes to Google's Gemini API with your own key, the Windows .exe is unsigned (SmartScreen will warn you), and a Python version is available. Repo: https://github.com/Spacexcx/pr-check