Backend
ProductWebBench: Evaluating Website Continuity Beyond Task Completion
arXiv cs.SE0 views
arXiv:2610.31118v1 Announce Type: new Abstract: Frontend coding agents are usually graded on one question: did the requested target get built. Real engineering changes carry a second obligation, because the rest of the site has to keep working. We call that second obligation website continuity, and introduce ProductWebBench, a benchmark that scores both on living frontend repositories: 320 Change tasks over existing code and 80 long-horizon Build tasks. Every task freezes a repository snapshot, staged requirements, browser evidence and deterministic checks before evaluation, and enters only once a reference passes every check and plausible-wrong ones fail. A simulated user reveals the requirements one stage at a time, and every check cleared earlier stays armed for the rest of the run. Across 8,737 runs from 13 models the two come apart: of the Change runs passing every requested-work check, 28.2% still fail a continuity check -- a blank route, horizontal overflow, console errors -- and 10.8% push a browser state they had already passed back to failing. Because the same checks also run on the untouched repository, each regression carries exact provenance, not a post-hoc label: on the 586 runs where both readings exist, 18.3% are caught breaking a verified state. Interaction is a further blind spot, failing in 29.5% of runs whose static DOM already passes. ProductWebBench therefore scores a patch against the living system it lands in, not the target alone.
Source:https://arxiv.org/abs/2610.31118PreviousTracing Hallucinations in LLM Reasoning for UML Class Diagram Generation: An Empirical Taxonomy and Mitigation StudyNextBeyond Accuracy: Disentangling Decomposition from Typed-State Quality Assurance in Multi-Agent LLM Pipelines
Related posts
I read all 308 open jobs at xAI: 54% are in its data center team, and most of them are in Memphis
0dev.to AI
Claude Code marketing analytics with an app-scoped key
0dev.to AI
I red-teamed my own support bot with 30 scenarios. It failed 7 — here's the harness.
0dev.to AI
The Log That Told Everyone the Password: Redacting Secrets in n8n Execution Logs
0dev.to WebDev
Your AI Portal Isn't Ready Until a Stranger Can Recover from Failure
0dev.to WebDev
API traffic monitoring: what to track, and how to set it up for a Node API
0dev.to Security