Results are aggregate; no developer, repo or commit is named. This is observational: "associated with", never "causes".

Question: are commits that disclose AI help (e.g. a Co-Authored-By: Claude trailer) reverted or followed by a bug-fix more often than comparable human-only commits?

I wrote down the hypotheses and definitions before pulling any data. The short answer: no clear difference, but the study is too small to say much.

Method in brief

  • Public GitHub APIs only, through my own authenticated gh. No repo was cloned and no code was run.
  • Repos with 50+ stars, not forks or archived, active in the last year.
  • Each AI-disclosed commit was matched 1:1 to a human-only commit from the same repo, within 30 days, with similar diff size, and from the same author when possible (82.5% of pairs).
  • 663 matched pairs, 378 repos, commits from 2026-03-12 to 2026-05-04.
  • Primary model: conditional logistic regression on the matched pairs.

Matching on size worked: median diff was 102 changed lines for AI-disclosed commits and 103 for controls.

Diff size distribution

Result 1: reverts are too rare to learn anything

Within 30 days, 1 of 663 AI-disclosed commits was reverted, versus 2 of 663 controls. Odds ratio 0.51, 95% CI 0.05 to 5.57. That interval spans "much safer" to "5x worse". I checked my revert detector against the 264k commits in the downloaded histories: about 0.2% of commits get reverted, so reverts are just rare. Detecting a 2x effect at that rate would take on the order of 10,000 pairs (rough calculation).

Result 2: the fix heuristic is mostly noise

My pre-registered "fix follow-up" rule: a later commit within 14 days, touching a same file, with fix|bug|hotfix|regression in the message. On a feasible 220-pair subset: OR 1.15 (0.75-1.77), consistent with no difference.

But I then read 100 flagged follow-ups and judged only about 30% to be plausible fixes (95% CI 22-40%). Common errors: shared package.json/README/lockfiles, a keyword in the body of a non-fix commit, merges of the same fix. So that "null" mostly reflects a noisy measurement. A stricter, exploratory variant gave the same picture (OR 1.16, 0.72-1.86).

Odds ratios

What went wrong, and what I changed

  • Sample shortfall: target was 2,000 treatment commits; I got 663 pairs. The 50-star filter removed about 93% of repos, and I stopped collection early for time.
  • Bug I caught: my first fix analysis silently dropped 39 pairs where a commit had no follow-up candidates. Fixed; the paper uses the corrected numbers.
  • Feasibility cut: the fix outcome only covers pairs with at most 20 candidate follow-ups, which favours quieter periods.

Threats to validity (the important ones)

  • Only disclosed AI use is visible, so this studies developers who disclose.
  • Controls may contain undisclosed AI code, which biases toward "no difference".
  • A revert is a weak quality signal.
  • The search sample is capped at 100 results per query-day and favours the latest commits of each day; it covers only 39 days.
  • One unblinded reviewer judged the heuristic; recall is unknown.

Takeaway

For this sample: no evidence of a difference, and little ability to detect one. A real answer needs about 10x more pairs and a validated way to link fixes to the commits they fix (issue links or SZZ-style tracing), plus some idea of how reliably people disclose.

Related reading (arXiv): On the Use of Agentic Coding, Agentic Much? Adoption of Coding Agents on GitHub.

Preprint-style write-up, not peer reviewed.

By Vyankatesh Rohokale