The Decoder ran a story on Saturday about a benchmark called Co-Design Bench, and the two headline numbers look contradictory until you read the method. The strongest of ten model endpoints meets 69.9% of a task's acceptance criteria on average, and fully completes 26.4% of tasks. Both numbers describe the same runs.
The paper comes from Alibaba's Token Hub group with researchers from Fudan, Zhejiang, CUHK, Shanghai Jiao Tong and NTU, posted to arXiv on October 26. What makes it worth an hour of your time is where the tasks come from. Not synthetic prompts. The authors curated 273 tasks from 37,463 deduplicated queries across 26,314 real user coding sessions, then wrote 2,331 hidden, expert-authored acceptance criteria for them.
Two numbers, one set of runs
Each task starts from a runnable web app, recovered and validated from the session logs, with a normalized user request attached. The agent edits the app. The rubric then grades criteria, not vibes.
The best endpoint, which The Decoder identifies as GPT-6 Astra, passes 69.9% of criteria per task on average. It fully passes 26.4% of tasks. Read those together: 73.6% of tasks keep at least one criterion unsatisfied. The agent usually does the bulk of the work and leaves at least one thing unfinished.
The track split is the interesting part. On Multimodal Editing, localized edits with optional image, audio, video or document inputs, the leader covers 75.4% of criteria and fully finishes 41.0% of tasks. On Repository Editing, coordinated changes across components, routes, state and data flow, it covers 66.7% and fully finishes 17.9%.
A 70% criterion coverage is not 70% progress toward done. The missing criteria sit at the end of dependency chains, and partial credit does not compile.
Where the tasks come from, and why that matters
The curation ranked candidates by structural difficulty, verifiability, and historical failure signals, specifically cases where users reformulated a request multiple times or rolled back changes. So this is a stress set built from work agents already found hard.
NxCode's read is the caveat worth repeating: criterion coverage and full task pass are different measurements, and neither is a measured rate of human cleanup across ordinary coding work. Do not quote 26.4% as "agents fail three out of four everyday tasks". The honest version: on hard, real-session revisions, even the best endpoint leaves something unfinished most of the time.
Judge-then-refute: the grading discipline worth stealing
The evaluation protocol is the part I keep thinking about. A first judge grades each criterion using source code and browser evidence. A second LLM verifier then challenges every positive verdict, hunting for stubs, hardcoded outputs, unreachable entry points and requirement mismatches. Visual claims require successful renders and screenshots. Interaction claims require recorded browser actions. Missing evidence aborts the evaluation.
The paper puts the principle plainly: a handler in source code alone cannot establish that users can reach and operate the requested functionality through the rendered interface.
That is exactly the review discipline most agent workflows lack. We read diffs. The agent writes a route handler, a component and a migration, the diff looks complete, and the criterion that fails is "the button on the settings page actually triggers the export". A probe that loads the app settles in seconds what a diff review cannot:
// acceptance probe: does the feature exist, or does it just read like it does await page.goto("http://localhost:3000/settings"); await page.click("text=Export"); const download = await page.waitForEvent("download"); // times out if the button is decorative If your verification step only greps the diff, you are grading with the judge and skipping the refuter.
The change-breadth cliff
One finding holds across all ten endpoints: criterion completion drops on repository edits that span three or four change types compared with one or two. Coordinated changes across components, routes, state and data flow are where partial delivery bites hardest.
The practical translation is boring and effective. Split wide requests before you hand them to the agent. One or two change types per task, sequenced, each with its own check. "Move the tax total to the summary card, keep the per-item breakdown in sync, and persist the currency preference" is three tasks wearing one sentence, and the third criterion is the one that will not survive contact with a single-shot prompt.
The harness is part of the score
Every endpoint ran in the same harness, Claude Code, capped at three hours and 120 turns per task. The authors still report harness sensitivity, and recommend reporting model and harness together. The Decoder notes Qwen3.8-27B scored 75 or 56 on larger overhauls depending on the environment.
This matches what Hugging Face measured last month with OpenEnv, where the same weights scored 62% under one harness and 33% under another. If you want that story, I wrote it up here: the same model scores 33% or 62% depending on the harness.
What this changes in my agent configs
Three things, all cheap:
- Acceptance criteria go into the task itself, as a short explicit list. Hidden rubrics are for benchmarks. In real work the agent should see the criteria, because the agent cannot chase a target it cannot read.
- The verification step loads the app and clicks the thing. Diff review catches syntax and style. It does not catch unreachable functionality.
- Wide requests get decomposed before dispatch, and the last criterion in a chain gets a human looking at it, because that is where the 73.6% lives.
Google's Elephant-Goldfish plugin, which I covered yesterday, pushes the same discipline one step earlier: catch the wrong assumption in a spec file before any code exists. Same instinct, different phase. Read that piece here.
Limits, stated plainly
The benchmark evaluates one fixed request per task. No follow-up questions, no mid-task corrections, which is not how real collaboration goes. The judges are LLMs, though two frontend experts spot-checked 200 task outputs and rescoring with other judge models moved the Qwen3.8-27B repository score by about 0.04 at most. The work is a preprint, not peer reviewed, and Alibaba builds Qwen while testing Qwen. The public subset covers 96 of the Multimodal Editing tasks and 521 criteria; the Repository Editing track is not public yet.
The gap itself is solid enough to act on. Not the 26.4%, which belongs to a stress set. The shape: agents deliver most of the criteria most of the time, and complete tasks only when the last criteria are cheap. Plan your reviews for that shape.
