AI & ML
Evaluating Governed LLM-Driven ML Experimentation: Lifecycle, Provenance, and Failure Metrics
arXiv cs.SE0 views
arXiv:2611.23678v1 Announce Type: cross Abstract: Autonomous agents that run ML experiments are usually judged by the best score they reach. That score says nothing about how it was reached: whether the agent stayed within the files it was allowed to edit, whether failed trials were kept or silently dropped, and whether each keep or discard decision can be traced to its evidence. We propose evaluating such systems on their experimental process as well as their outcome, and we build a governed harness that makes this possible. A control plane, not the LLM that proposes changes, owns each trial's lifecycle, checks its edit scope, decides keep or discard, and appends one record per trial to an append-only ledger. Governance metrics such as the invalid-action rate, the repeated-bad rate and provenance completeness are computed from that ledger. Across six ML tasks, from synthetic logistic regression to language-model training, and 1,445 trials, the same contract applied unchanged. The ledger exposed failures that a task score hides: a no-memory arm in which all 120 trials failed, a benchmark adapter whose bounded proposals ran out, and a reset failure that reproducibility hashes revealed, after which the affected campaigns were rerun. In a matched counterfactual, a loop that logs only successful trials left no record of 43 of 50 trials, while the governed ledger recorded all 50. Memory ablations show that memory's effect on repeated bad proposals depends on the task. We release the ledgers of all 1,445 trials, so that every number reported for them can be recomputed.
Source:https://arxiv.org/abs/2611.23678PreviousRefinementGuard: Counterexample-Directed Semantic Qualification for Hidden Agent-Tool ExecutionNextValidating Concurrent Correctness Witnesses with Ghosts by Thread-Modular Abstract Interpretation
Related posts
How do Balkan freelancers get paid by US clients without Payoneer?
0dev.to
Fuzzy name matching for sanctions screening: normalisation, transliteration and Jaro-Winkler
0dev.to
The Claude Architect Professional exam changes on 7 December - Here is what's different
0dev.to
SEO Isn't Enough Anymore. Meet GEO.
0dev.to

Prompt Engineering vs. Context Engineering vs. Harness Engineering: What’s the Difference?
0dev.to
P_SAPEA Exam Practice Questions: The Comparisons That Decide SAP Enterprise Architect Scenarios
0dev.to