AI & ML
We Fixed the Scorer First, and Cluster Correction Killed Our Own Planted-Defect Result in AI Code Review
arXiv cs.SE0 views
arXiv:2611.21635v1 Announce Type: new Abstract: Judging an AI code-review pipeline only by how much it catches hides an axis: what it dropped. Stronger selection raises precision for free; the cost never reaches the report. We ran four arms on the same eight cases, one scorer, asking whether planted-defect detection can rise without raising the claim error rate. Three repetitions are nested in eight cases, so p is reported per contrast and twice: under an exact permutation test over cases (2^8 arrangements) the two contrasts publishable under independence (Fisher p=0.017, 0.019) move to 0.0625 and 0.109, and after Holm correction no contrast separates. This is "not observable with this design," not "no effect": the sign test needs six discordant cases in one direction, its realized type-I rate under the null is 0.0078, and 80% power needs a per-case win probability near 0.80; closing it takes 12-25 cases, not repetitions. Within a run, a second pass never removes what the first found (0/24, 0/24 paired losses) while its gains (+3, +2) fall in one case; baselines on byte-identical prompts differed 15/24 vs 20/24, so run-to-run variance exceeds the effect. The claim error rate is 0.0% (0/178), with no upper bound on the rate claimed; scorer discrimination was measured with pre-registered positive controls (12/12 fabricated claims rejected, 0 false rejections). We retract our original causal explanation and report the instrument defects we hit. The contributions are not the numbers: (1) fixing the instrument before seeing results, then killing our own main effect with it; (2) the identity eps-hat = eps-hat(pi): a reported error rate is computed only on the subset a retention policy kept, silent about what was dropped, so it does not identify the quantity an irreversible gate needs; no improvement in the number promotes an automated verdict to publish, delete, or history-rewrite surfaces; a mechanized refusal enforces this.
Source:https://arxiv.org/abs/2611.21635PreviousTrusting Trust in AI Software Engineering Agents: The Why, What, and How?NextA Survey of Modeling Languages for Automotive Digital Twins
Related posts
A Canva alternative for Instagram posts, stories and carousels with your brand applied in one click
0dev.to Design
What Might the Next Generation of AI Look Like?
0dev.to Discuss
NVIDIA Q3 FY27 Earnings: $115.8B Revenue, $390B Supply Gamble and the Post-Jensen AI Infrastructure Test
0dev.to Discuss
Do developers actually think about code energy efficiency, or is it just about cloud bills?
0dev.to Discuss
Best AI Humanizer Tools for SEO Articles and Long-Form Blog Posts
0dev.to Productivity
Beyond Notes: 10 Hidden Power‑Ups in iPhone’s Notes App
0dev.to Productivity