AI & ML
Programmable Distributions of Semantics-Preserving Variants for Evaluating Code Agents
arXiv cs.SE0 views
arXiv:2608.18389v2 Announce Type: replace-cross Abstract: Code agents are increasingly deployed to resolve real software issues, yet their reliability under superficial code variations remains poorly understood. We evaluate whether coding agents that repair repository-level issues remain reliable when the surrounding codebase is rewritten into a semantically equivalent form. We sample semantically equivalent variants of a repository from an explicitly defined distribution and compare the agent's resolve rate on them with that on the original. Different distributions reveal different aspects of an agent, such as its sensitivity to routine refactoring or to misleading code near the location of a fix. We develop VDL, a probabilistic programming language for expressing such distributions, whose programs randomly apply semantics-preserving transformations (SPTs) to a repository. For our evaluation, we use it to define a generic distribution that is not targeted at any particular failure mode. We evaluate two agentic scaffolds (mini-SWE agent and OpenCode), each backed by one of four frontier models (Claude Opus 4.5, Kimi K2.5, MiniMax M2.5, and Qwen 3.6-27B), on instances from SWE-bench Verified and SWE-bench Pro. Resolve rates drop by up to 6.7 percentage points, significantly in 6 of 16 configurations. No ranking of models by robustness holds across scaffolds, revealing a jagged robustness frontier.
Source:https://arxiv.org/abs/2608.18389PreviousThe $\alpha$-Limit Problem: Convergence of a Linear Degenerate Interface Transmission ProblemNextCross-Benchmark Transfer from RL on Agentic Coding Tasks
Related posts

94% of Singapore Small Business Owners Surveyed Say More AI Guidance Is Needed
0Meta Newsroom
Nashville, Tennessee Real Estate Dataset: 7,436 Listings, Excel File (Nov 10, 2026)
0dev.to Data
A Hundred and Eighty Six Thousand Pounds of Stock Had Left One Branch and Never Arrived at Another
0dev.to Data
Đánh giá thông số 20 tiền đạo xuất sắc nhất bóng đá châu Âu thế kỷ 21 từ BBC Sport
0dev.to Data
Thống Kê SHB Đà Nẵng: Bài Toán Phòng Ngự Chờ HLV Lee Young-jin Hóa Giải
0dev.to Data
Alibaba's Qwen4-Max Is Here — and It's Gunning for Claude's Coding Crown
0dev.to News