AI & ML
VeeReview: Measuring Build-Verify Pairs in Agent-Authored Pull Requests from a V-Model Perspective
arXiv cs.SE0 views
arXiv:2610.29143v1 Announce Type: new Abstract: Context: Reviewing a pull request requires verification evidence that addresses the proposed change. Coding agent benchmarks and studies of pull request outcomes do not directly assess this relationship. Objective: We test whether the V-Model's build and verify pairs can measure verification evidence in pull requests and support comparisons between agent-authored and human-authored changes. Method: VeeReview classifies four build and verify pairs using repository signals and a language-model judge. Three pairs use model judgements constrained by deterministic signals, and a rule over the diff and repository tree decides Module Design-Unit Test. We evaluated VeeReview against a 25-pull request reference set labelled by two raters and applied it to 1,454 matched pairs of agent and human pull requests from 741 repositories. Results: VeeReview assigned lower build-only rates to agent-authored pull requests on all four pairs, contrary to the preregistered hypothesis (mixed-effects odds ratios 0.52 to 0.86). Architecture-Integration was not significant. Standardization for linked issues, shipped tests and size reduced matched-rate differences by 34% to 60%, and the rule-decided pair's standardized interval included zero. The instrument did not meet its preregistered calibration criterion: agreement with adjudicated labels was 4% to 60% against a required 80%, and inter-rater agreement was weak (kappa = 0.154). Conclusions: The rule-decided pair shows that a process measure judged by a language model partly reflects how a pull request is presented rather than the verification work behind it. We report the cohort differences as observations made with this instrument, and we argue that such measures need a mechanically decided component and a validated human reference before they can support comparisons of verification practice.
Source:https://arxiv.org/abs/2610.29143PreviousSymmetry-Oriented Testing Framework for Variational Quantum AlgorithmsNextVERDICT: Multi-Agent Parallel Modular Verification
Related posts

This 'SMS of Death' exploit can remotely crash unpatched Android phones — without the user interacting with it in any way
0TechRadar

OpenAI wants to rebrand data centers as 'compute campuses' to try and win people over
0TechRadar

Putin demands Russian data centers supply capacity to the state in wake of Ukraine drone attacks
0Tom's Hardware

OpenAI considers rebranding data centers to 'AI compute campuses" as data center backlash intensifies
0Tom's Hardware
What tech entrepreneurs can teach scientists about failure
0Nature News

X for iPhone and iPad just added over a dozen new app icon options
09to5Mac