Frontend
Time-Consistent Evaluation of Hierarchical Code Graphs for SWE Agents
arXiv cs.SE0 views
arXiv:2611.17388v1 Announce Type: new Abstract: Evaluating repository-aware software engineering (SWE) agents requires strict separation between pre-existing repository knowledge and future code evolution, yet standard benchmarks frequently suffer from temporal contamination and prompt-level solution leakage. Building on the time-consistent benchmark protocol of Sun (2026), we enforce strict zero-leakage isolation across DragonFly in C++ and React in JavaScript/TypeScript, locking repository snapshots and knowledge graphs at historical cutoff commits $T_0$ and evaluating 100 post-$T_0$ pull requests across four prompt granularities: Minimal, Concise, Contextual, and Guided. Under this zero-leakage protocol, we conduct a matched evaluation of three context-augmentation regimes: (1) an unaugmented Baseline, (2) a flat syntax-level Tree-sitter AST graph, and (3) ServiceExplorer, a deterministic two-level hierarchical dependency graph capturing Level-1 system/package containers and Level-2 architectural modules. Across 7,200 paired evaluations spanning six foundation models across the MAI-Code Flash, Claude, and GPT-Sol families, ServiceExplorer consistently outperforms both Baseline and Tree-sitter AST across every model, repository, and prompt granularity ($+15.6\%$ to $+86.2\%$ relative $F_1$ lift, $p < 10^{-4}$), while tripling exact full-PR localization on DragonFly. Crucially, equipping an efficiency-tier or prior-generation model with ServiceExplorer consistently surpasses the unaugmented Baseline $F_1$ of the next-generation model across all evaluated families.
Source:https://arxiv.org/abs/2611.17388PreviousProdPilot: A Layered Model Context Protocol System for Automated Production Readiness and Deployment of Vibe-Coded ProjectsNextInterpreting Coding Agents Through Their Behavioral Trajectories
Related posts
You Spent the Outage Fixing the One Red Light and Nobody Was Watching the Rest
0dev.to
How to Trim Audio for a Virtual Team Building Video — Free, in Your Browser
0dev.to Tutorial
How to test a video your AI agent rendered: From seek bugs to the final MP4
0dev.to Tutorial
How to test AI-Rendered Videos Before You Ship?
0dev.to Tutorial

How to Read a Prescription: What the Abbreviations on Your Medicine Label Actually Mean
0dev.to Beginners
How I Built My Own Desktop Voice Assistant (Jarvis) Using Python as a Class 9 Student
0dev.to Beginners