AI & ML
OneCodeReviewBench: Typed Decisions for Code Review and Search
arXiv cs.SE0 views
arXiv:2610.33389v1 Announce Type: new Abstract: Code review requires finding functional bugs, memory-safety errors, security weaknesses, and performance problems across languages and software stacks. Tools can express bounded parts of this work as decisions about supplied code. We present OneCodeReviewBench, 1,968 synthetic cases across 12 programming languages and YAML, with Boolean, categorical, and ordered answers. Explicit contracts specify runtime assumptions, source scope, and answer criteria; separate evidence records support reference review. One author reviewed 150 cases; two additional engineers reviewed 100 of those cases. We compare ten model--adapter systems on the same cases and contracts: OpenAI Decisions reaches 81.3% Boolean balanced accuracy; Kev-27B reaches 76.8% Choice accuracy and Score mean absolute error of 0.367. Excluding two families with incomplete contracts preserves these leaders. Additional comparisons cover 586 cases with raw symbol search and 40 with a bounded Python import resolver; the resolver matches all 40 references within its supported profile. A versioned artifact reproduces these analyses and retains the 85 cases with matching engineer answers; the reported review subset excludes two incomplete-contract cases. Remaining reviewer disagreements and repaired inputs require resolution; repository-scale transfer and matched latency/cost evaluation remain future work.
Source:https://arxiv.org/abs/2610.33389PreviousEVM-Anchored Provenance for AI-Assisted Code Evolution: A Smoke-Scale Deployment ReportNextDevelopers as Managers of AI Agents: A Workplace Shadowing Study
Related posts
How To Recover An Affiliate Site After A Google Spam Update
0dev.to Beginners
When a PDF Export Turns Every Page Into a Picture, Define the Text Contract First
0dev.to Testing
Software Testing for Beginners: Why Working Code Is Not Always Correct Code
0dev.to Testing
I Graded 120 Agent Runs. Enforcement Bought the Paperwork, Not the Fix
0dev.to Testing
AI Agent Security Guide
0dev.to Kubernetes
Hydra v1: new DOI and the complete DOI registry (2026-11-02)
0dev.to OpenSource