AI & ML
FeatureLoop: Benchmarking Coding Agents under Evolving Requirements
arXiv cs.SE0 views
arXiv:2610.35488v1 Announce Type: new Abstract: Software development with coding agents often begins with a rough request that users refine after trying the implementation. Final correctness alone does not capture the repeated guidance this process requires. We introduce FeatureLoop, a benchmark of 61 human-reviewed tasks from 34 repositories across five primary programming languages and seven software forms. Each task follows one development goal in a persistent workspace through evolution feedback, which adds or clarifies requirements, and completion feedback, which reports failures of stated requirements. Cumulative executable checks track new behavior, repair, and preservation of earlier functionality. We evaluate 12 frontier models and find that the strongest completes 72.1% of tasks with both feedback types. Across models, however, only 15.4% of observed failed stages recover within two corrective revisions. Further analysis reveals incomplete repairs, regressions, and model-specific differences in feedback use and token efficiency, identifying obstacles to reliable, low-intervention development.
Source:https://arxiv.org/abs/2610.35488PreviousUnderstanding Python Linters through a Comprehensive and Systematic Empirical StudyNextThe Hidden Costs of Forgetting: A Study of Trajectory Deletion in Software Engineering Agents
Related posts
Best Dyson Vacuum Deals (2026): Dyson Spot+Scrub Ai, Dyson V16 Piston Animal
0Wired
CPU, GPU, TPU, NPU: Who Will Power the Future of Computing?
0dev.to

Lower loss, worse stories: small NCMs trained on TinyStories
0dev.to

I Built LLM Explained: 43 Interactive Pages That Take You From Tokens to AI Agents
0dev.to OpenSource
MIT CSAIL: Astra edita 0,016% de pesos de Qwen3.5-0.8B y sube MATH-500 a 48,1%
0dev.to Programming

Gemma 4 in Rust on TorchTPU: A Native TPU Rig With PyO3 and MCP
0dev.to Rust