AI & ML
ParanoiaEval: Benchmarking Unnecessary Defensive Work in Agentic Coding
arXiv cs.SE0 views
arXiv:2610.08662v1 Announce Type: cross Abstract: As coding agents increasingly undertake real-world work autonomously, judging whether their risk treatments are warranted has become important. Existing work evaluates related agent behaviors from separate perspectives, but lacks a systematic framework for unifying these behaviors. To bridge this gap, we introduce ParanoiaEval, the first benchmark for unified evaluation of risk-treatment capabilities in coding agents. Grounded in the well-established Avoidance-Transfer-Mitigation-Acceptance framework in software engineering risk management, ParanoiaEval operationalizes its 4 fundamental treatments for coding-agent settings and contains 200 evidence-controlled repository-level task pairs, each differing only in treatment-defining evidence. We further introduce dedicated metrics for risk-treatment violations and evidence responsiveness, using a human-calibrated agentic judge for reliable evaluation. Large-scale experiments on 8 representative models and a post-hoc human study reveal that (I) unnecessary risk treatment occurs in 11.2%-58.7% of runs despite explicit evidence, with substantial variation across agent configurations; (II) stronger task capability does not ensure more appropriate risk treatment, while treatment violations substantially harm developers' experience, establishing risk treatment as an independent capability dimension; and (III) agents exhibit systematic patterns consistent with established risk-management findings, suggesting that knowledge from human practice can guide the diagnosis and improvement of this capability.
Source:https://arxiv.org/abs/2610.08662PreviousRecursive Game Creator: An Agentic Product-Level Experience-Oriented Game HarnessNextTheory building in software engineering: Operationalization
Related posts
Launch a Business for Under $500: Real Odds, 4 Founders
0dev.to AI
Google, Microsoft, OpenAI and Anthropic Back a Shared AI Training Opt-Out Signal
0dev.to AI
Did AI Decode the Voynich Manuscript? Claims Tested
0dev.to AI
AIGB 解决 Teletype.in 群组管理痛点
0dev.to AI

Qual a melhor arquitetura de dados para a sua aplicação de IA?
0dev.to AI
Connect a Stateless Streamable HTTP MCP Server to Claude, ChatGPT, and Gemini for Tour Route Research
0dev.to WebDev