AI & ML
Variance-adaptive confidence regions for fixed-benchmark binomial mixtures, with application to LLM evaluation
arXiv stat.ML0 views
arXiv:2611.21266v2 Announce Type: replace-cross Abstract: We construct confidence intervals for linear functionals $\theta(F_m)=\int h\,dF_m$ of the empirical law $F_m$ of success probabilities $p_1,\dots,p_m$, from independent counts $c_i\sim\mathrm{Bin}(n,p_i)$, as for a language model run $n$ times per problem. Coverage is at least $1-\alpha$ for every $n$, $m$ and fixed $(p_i)$, simultaneously for all bounded $h$. The variance-adaptive (VA) region tests each candidate mixing law with the conditional variance of the tail counts it implies. This variance is linear in the law, so the region is a linear program, and cell envelopes keep it valid off-grid. Regions calibrated for iid sampling target a super-population law and are conservative for $\theta(F_m)$. With all $n$ tail counts, fixed $n$, continuous $h$ and a grid refined as $m\to\infty$, the identification part of VA's width is first-order worst-case optimal, as for any localization with the same index set. Under a rule fixed before the main comparator runs, VA was narrower than a two-stage plug-in band at every $n$ (median width ratio 0.60 at $n=128$), but not than the asymptotic, pointwise AMARI interval at $n=128$; on real rollout pools it was wider than both pointwise and Bonferroni-adjusted AMARI at $n\le128$. For $P(p>0)$ and pass@$k$ with $k\gg n$ the data support only lower bounds.
Source:https://arxiv.org/abs/2611.21266PreviousTight Lower Bounds for Differentially Private Continual CountingNextA Large Gap between Galerkin Errors with and without Mesh-Uniform Regularity for the KdV Equation with Boundary Dissipation
Related posts

Debugging Claude Fable 5.2 Code Art: Truncated SVG, Broken Canvas, Working Render
0dev.to
Hydra · Coeziune OM–IA — the complete DOI registry (2026-11-18): 76 published works
0dev.to OpenSource
Your Rust Code Sets an Environment Variable and the 2024 Edition Calls That Unsafe
0dev.to Rust
Your Go Function Returned a Nil Error and the Caller Saw an Error Anyway
0dev.to Go
Kengele: an alarm for when an activist goes missing, that nobody can switch off
0dev.to Go
How OpenAI scales PostgreSQL to 800 million users
0dev.to Database