AI & ML
Regime-Conditioned Evaluation in Multi-Context Bayesian Optimization
arXiv stat.ML0 views
arXiv:2605.04895v2 Announce Type: replace-cross Abstract: Published transfer-BO comparisons often estimate an average treatment effect of acquisition choice over hidden regime variables, while practitioners need the conditional effect for their specific prior quality, budget ratio, and metric. An audit of 40 transfer-BO papers from NeurIPS, ICML, ICLR, AISTATS, UAI, TMLR, JMLR, and AutoML-Conf (2022-2025) finds that 98% never vary B/|A| as a controlled axis. On the same GDSC2 benchmark, changing only the budget reverses the ranking: at B=50, Greedy outperforms UCB by 0.050 Hit@1, while at B=100, UCB outperforms Greedy by 0.035. We capture this transition with the Portable Regime Score PRS=(B/|A|)(1-rho), where rho is the prior rank correlation and can be estimated from pilot contexts before the main comparison. Across 79 conditions spanning chemistry, drug-response biology, and HPO, a hierarchical model gives beta=0.50 (p=1.1e-9), and 19% of conditions fall in an equivalence zone where |advantage|<0.01 Hit@1. In five published reversal cases, PRS predicts the winner from pre-comparison observables. A No-Free-Leaderboard proposition explains why unconditional rankings are unstable: when CATE changes sign across regimes, the reported ATE becomes a function of benchmark mixture. RegimePlanner, which estimates rho online and switches acquisition accordingly, wins all 16 HPO-B search spaces at B=100 and exceeds the matched {Greedy,UCB} per-context oracle on GDSC2 by 18%. Pre-registered predictions achieve 27/40=67.5% overall accuracy and above 90% within EMA prior families. The practical protocol is simple: report B/|A|, rho, K, and metric alongside any claimed acquisition advantage.
Source:https://arxiv.org/abs/2605.04895PreviousEmpowering Bayes with sources of uncertainty: from confidence modelling to sparse estimationNextHuman-Centered Learning Mechanics: Observable Entropy Control under Protected-Risk Constraints
Related posts
BeeTalk 群主必备:AI 群管机器人 AIGB 免费解决垃圾信息问题
0dev.to AI
I Open-Sourced a Free YouTube Shorts Factory. The Money Isn't in the Shorts.
0dev.to AI
Priced per TB/s, the H200 beats the H100 at 6 of 7 providers
0dev.to Python
Daily Dose of DevOps — Telemetry design: for platform operations teams
0dev.to DevOps

Israeli startup 42AI, which is developing a "behavioral world model" that tries to predict human behavior, emerges from stealth with a $62.5M seed (Ram Iyer/TechCrunch)
0Techmeme

Anthropic confirms new Claude Code limits "to steer it away from recreating someone else's specific game", saying IP infringement violates its usage policy (Bruno Ferreira/Tom's Hardware)
0Techmeme