AI & ML
Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3
arXiv cs.SE0 views
arXiv:2610.30499v1 Announce Type: cross Abstract: A generally intelligent system acquires new skills efficiently without requiring humans to redesign its software scaffold for every new task. Yet language models rarely act alone. A harness is a symbolic program that orchestrates calls to the model, handling perception, memory, tool use, recovery, and inference allocation on the model's behalf. ARC-AGI-3 is an interactive benchmark in which agents must explore unfamiliar environments, figure out their goals on the fly, and keep learning as they go. On it, human-steered systems already report ceiling scores across the public games, so we know effective designs exist. We ask whether frontier models can build and improve such a harness on their own. Through a fixed coding bootstrap, each model constructs a complete task harness, acts through that harness on five development games, and revises the entire codebase twice. Every version is evaluated on twenty games that never influence adaptation. We measure task performance with Relative Human Action Efficiency (RHAE), which combines level completion with action efficiency relative to human baselines, and record dollar-denominated inference cost as a practical resource measure. Across 27 lineages, harness performance ranges from zero to near-ceiling scores. Two revisions raise mean held-out RHAE by only 2.80 points (95% interval [-1.44,7.35]). Development gains overstate held-out improvement, and reductions in inference cost can come at the expense of task performance. Human-steered systems demonstrate further headroom. These findings reveal a gap between autonomous harness construction and reliable adaptation under this two-revision protocol: even strong task performance does not ensure that subsequent revisions generalize or preserve existing capabilities.
Source:https://arxiv.org/abs/2610.30499PreviousFrom Prose to Properties: Testing Research Claims with ARATest for Agent-Native ScienceNextIRAH: Hierarchical Agentic Tree Search for Retrieval Pipeline Optimization
Related posts
We checked three LLM-written paper explainers against the PDFs. Two had numbers the papers never report.
0dev.to Machine Learning
Should You Rent AI or Run Your Own?
0dev.to Machine Learning
Who's hiring this week: only 1 in 30 new tech jobs is junior, and 1 in 6 'AI jobs' is AI-training work (Oct 26-Nov 1, 2026)
0dev.to Career
ClarityHope – An Online Counselling Platform for Accessible Support
0dev.to Career
The 3 documents to have before you start any freelance project (free deposit clause inside)
0dev.to Career
How I Diagnose a Slow, Loud, or Overheating PC: A Practical Checklist
0dev.to Beginners