An AI coding agent’s overall acceptance rate can hide where it helps and where it adds review work. To evaluate one safely, separate work by task type, define acceptance criteria, and track outcomes over time.
A 2026 study of 7,156 pull requests found that task category often mattered more than which agent produced the change. Here’s how to apply that finding to an engineering evaluation.
What the study measured
The paper Comparing AI Coding Agents: A Task-Stratified Analysis of Pull Request Acceptance compared five coding agents across nine task categories and tracked acceptance rates over 32 weeks.
Acceptance rate indicates whether submitted pull requests were accepted. It does not, by itself, measure code quality, security, maintainability, or developer productivity.
The task-level differences were substantial: documentation had an 82.1% acceptance rate, compared with 66.1% for new features—a gap of about 16 percentage points. That difference was larger than the typical gap between agents within many categories.
Why task category matters
Documentation and feature work need different evaluations
Documentation changes often have a narrower scope and more visible review criteria. In the study, Claude Code led the documentation category with a 92.3% acceptance rate.
Feature development had a lower acceptance rate of 66.1%. Requirements interpretation, dependency choices, testing, and coordination all give a plausible-looking change more ways to miss the intended behavior. A strong result on documentation therefore does not establish that an agent is a good fit for feature work.
Category leaders were not the same
OpenAI Codex had consistently strong results across all nine categories, with acceptance rates ranging from 59.6% to 88.6%; the paper reports statistically significant advantages over other agents in several categories. But the category leaders varied: Claude Code performed best for documentation and feature tasks in the reported results, while Cursor led the fix category at 80.4%.
Devin was the only agent with a consistent positive acceptance trend during the 32-week observation period, increasing by 0.77% per week. The other agents were described as largely stable. These are historical results from one dataset and time window, not a guarantee of performance in another repository or under different review standards.
Build a task-stratified benchmark
- Choose representative task categories. Include work your team actually expects agents to handle, such as documentation, bug fixes, feature development, refactoring, and testing. The study covered nine categories; your benchmark should reflect your backlog.
- Define acceptance criteria before the task starts. State the objective, boundaries, relevant files or services, expected behavior, and required validation. Keep the requirements connected to the implementation and test evidence.
- Record results separately by category. Calculate acceptance rates for each task type rather than combining every pull request into one score. A blended average can conceal weak results in a particular kind of work.
- Track effort and quality signals alongside acceptance. Record review time, rework, defect escape, test failures, and the amount of human editing needed before acceptance. These are useful evaluation measures, but they are additional to the study’s reported acceptance metric.
- Repeat the evaluation. Record the agent version, task category, repository context, and acceptance criteria. Reassess periodically as models, prompts, repository conventions, and review practices change.
Connect agent output to the delivery workflow
A generated patch is only one part of a change. Reviewers still need to know why the work is required, which requirements apply, who owns the review, whether testing is complete, and what happens if the change is rejected.
Use a traceable workflow that links the request to its requirements, implementation, review, test evidence, and outcome. For higher-risk feature work, make review ownership and required checks explicit. If a change is rejected, record the reason and the steps needed to revise it. These controls help reviewers judge the change against its intent instead of reconstructing context from scattered conversations and documents.
Verify whether an agent is useful for your team
Compare results within each task category and include the full cost of getting a change accepted: review time, revisions, rejected pull requests, and post-release issues, as well as acceptance rate. Apply the same acceptance criteria consistently, and keep the review and testing controls appropriate to the work.
The study’s findings are a reason to evaluate agents by task, not a universal ranking to copy. A team’s own benchmark is needed to determine whether an agent fits its repositories, requirements, and review standards.
