AI & ML
DRAFT-Bench: Evaluating Frontier Agents Across the Professional Design Lifecycle
arXiv cs.SE0 views
arXiv:2611.08933v1 Announce Type: new Abstract: Can frontier general-purpose agents understand client intent, explore creative directions, execute revisions, and prepare production deliverables? We introduce DRAFT-Bench (Design Reliability Across Feedback and Tasks), a lifecycle-oriented evaluation of requirement elicitation, creative exploration, constrained editing, iterative revision, and production handoff. Its Requirement-to-Artifact framework links evolving requirements, protected properties, artifact versions, and delivery obligations. We compare native Codex/Astra and Claude Code across 35 design briefs under 43 conditions, alongside a separate 40-scenario RFQ track. Astra completes 43/43 design conditions and Claude 35/43. All scored core runs receive full structural credit, yet on the 28 core conditions scored for both systems, semantic checks flag violations in 11 for Astra and 5 for Claude; these are model judgments, not expert-confirmed failure rates. RFQ coverage is high, but success ordering changes across cohorts. SKU expansion, revision dependencies, and White ink files expose ambiguity among unmet obligations, missing inputs, and evaluator interpretations. These findings separate execution, requirement compliance, and professional acceptance. Unpaired tracks and different tools preclude claims of RFQ-to-delivery success, factory acceptance, or a pure model ranking.
Source:https://arxiv.org/abs/2611.08933PreviousLLM-Augmented Flow for Project Conformation and Task Complexity Evaluation Based on the TLM FrameworkNextMongoBench: Do LLMs Know What a MongoDB Query Will Do?
Related posts
Our Route Planner Sent Crane Lorries to the Middle of Each Postcode and Some Rural Postcodes Are Miles Wide
0dev.to Data
How much of GPT-6 Astra's weight-editing demo is the harness?
0dev.to Machine Learning
How Pet Groomers Can Track Appointments, Pet Notes and Rebookings
0dev.to Tutorial
Symmetrical Components in Plain English: Fortescue Without the Fog
0dev.to Tutorial
When P2P Pays You and the Account Freezes Anyway
0dev.to Career
Your Company Will Pay for the Course if You Agree to Pay It Back When You Leave
0dev.to Career