Backend
Knowing When to Stop: The Evidence-Action Gap in Coding Agents
arXiv cs.SE0 views
arXiv:2610.31895v1 Announce Type: new Abstract: Coding agents are trained to complete tasks, and they keep trying even when the environment makes completion impossible. The evidence that only outside help can restore progress usually appears within a few steps, yet agents rarely act on it: they over-persist, attempt undesired workarounds, and claim success the record does not support -- a phenomenon we call evidence-action gap. To measure it, we introduce ImpasseBench, which injects twelve realistic environment blockers into otherwise solvable repair tasks and marks, in each trajectory, the step at which the collected evidence becomes sufficient. Scoring actions relative to that step separates appropriate effort from over-persistence and premature handoff. Across eight frontier models and 19,371 trajectories, we find that nearly every run that needs outside help collects sufficient evidence, yet a third still exhaust their budget, and every unsafe request we detect comes after that point. Fortunately, an escalation tool, or an external monitor for models that ignore the tool, already guards against much of this behavior: the tool cuts budget exhaustion from 34.0% to 6.7%, and the monitor nearly triples task success after environment restoration. Even then, a fired escalation is not yet a useful handoff: tickets reliably name the blocker but often omit the agent's own progress and verification status. Taken together, we argue that knowing when to stop therefore needs explicit support, both in the tool that lets an agent hand off and in the report it leaves behind.
Source:https://arxiv.org/abs/2610.31895PreviousSWE-Clar: Can Coding Agents Clarify Incomplete Client Requirements and Deliver Greenfield Software?NextAre LLM-Generated Formal Specifications Faithful to Their Requirements? A Study on Bimodal Formal Specification Generation
Related posts

Nibnib says 4bpp. The file still has a header.
0dev.to Rust
I counted words in the same 100 MiB file with ten programs: straightforward and tuned versions in Rust, Go, Node and Python, plus two-thread Rust and Go. Read the full article : https://medium.com/git
0dev.to Rust
Redis 8.12 RC2's SEXPIRE command cuts a set membership check from two round trips to one
0dev.to Database
Does AbortSignal Cancel a Node.js Database Query? 4 Drivers Tested
0dev.to Database
I read all 308 open jobs at xAI: 54% are in its data center team, and most of them are in Memphis
0dev.to AI
Claude Code marketing analytics with an app-scoped key
0dev.to AI