Backend
Evaluating Exact Output and Checkpoint-State Prediction in Real Programs
arXiv cs.SE0 views
arXiv:2610.11889v1 Announce Type: new Abstract: We present a benchmark for predicting final output and checkpoint state from source and input alone. It extends CRUXEval-style output prediction with paired shorter- and longer-trace inputs and checkpoints inside and after a loop. The benchmark contains 400 cases from 371 Python and C++ programs, evaluated under seven settings from four model families without tools or code execution. Of 11,200 planned predictions, 11,151 produced gradable responses. Reasoning-enabled settings outperform their off counterparts by 33.1 to 55.2 percentage points on completed responses. The strongest setting scores 93.0% on shorter-trace final output, 77.0% on longer-trace final output, and 65.5% and 63.5% on the two state tasks; these scores also hold when missing responses count as wrong. Across 2,397 matched Python comparisons with identical source, changing to the longer-trace input yields 528 correct-to-wrong changes and 147 reversals. Source-clustered analyses preserve this accuracy gap, while adjusted Python models give no evidence of a positive incremental association between cumulative state load and error. Changed inputs and checkpoint tasks alter several factors together, so the gaps do not isolate trace length or an internal state-tracking mechanism. The benchmark exposes errors hidden by short-output scores alone.
Source:https://arxiv.org/abs/2610.11889PreviousTrajectory-Guided Fault Localization for Agent Skill EvolutionNextCan LLMs Fix It Without Code? Toward Automated Verification of No-Code Bug Fixes
Related posts
The fix for an empty state isn't filling it
0dev.to UX
Five Things That Make Dependabot Painful for Terraform
0dev.to GitHub
Tracking Roblox player counts: what the public data actually gives you
0dev.to Data
Telegram gold signals this week: results to Oct 30
0dev.to Data
Google Trending Searches Today in 10 Countries (Nov 02, 2026)
0dev.to Data
Custom-Made Furniture in India: Sizes, Budget, Timeline and What to Check Before Ordering
0dev.to Design