AI & ML
Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks
arXiv stat.ML0 views
arXiv:2605.10395v2 Announce Type: replace Abstract: We study the information-theoretic limits of learning a one-hidden-layer teacher network with hierarchical features from noisy queries, in the context of knowledge transfer to a smaller student model. We work in the high-dimensional regime where the teacher width $k$ scales linearly with the input dimension $d$ -- a setting that captures large-but-finite-width networks and has only recently become analytically tractable. Using a heuristic leave-one-out decoupling argument, validated numerically throughout, we derive asymptotically sharp characterizations of the Bayes-optimal generalization error and individual feature overlaps via a system of closed fixed-point equations. These equations reveal that feature learnability is governed by a sequence of sharp phase transitions: as data grows, teacher features become recoverable sequentially, each through a discontinuous jump in overlap. This sequential acquisition underlies a precise notion of \textit{effective width} $k_c$ -- the number of learnable features at a given data budget $n$ -- which unifies two distinct scaling regimes: a feature-learning regime in which the Bayes-optimal generalization error $\varepsilon^{\rm BO}$ scales as $ n^{1/(2\beta)-1}$, and a refinement regime in which it scales as $n^{-1}$, where $\beta>1/2$ is the exponent of the power-law feature hierarchy. Both laws collapse to the single relation $\varepsilon^{\rm BO}=\Theta(k_c d/n)$. We further show empirically that a student trained with \textsc{Adam} near the effective width $k_c$ achieves these optimal scaling laws (up to a small algorithmic gap), and provide an information-theoretic account of the associated scaling in model size.
Source:https://arxiv.org/abs/2605.10395PreviousA Cherry-Picking Approach to Large Load Shaping for More Effective Carbon ReductionNextKernel-based guarantees for nonlinear parametric models in Bayesian optimization
Related posts
Voyager 1 at one light-day: Historian revisits Golden Record's forgotten details
0Ars Technica

Grand Theft Auto VI has arrived
0The Verge

Suno now lets you make AI videos for your AI music
0The Verge
India’s Humyn Labs raises $47M to feed U.S. robotics companies the data they need
0TechCrunch
Felicis leads $7.5M bet on Unsiloed AI to make enterprise documents usable by AI agents
0TechCrunch

GPT-6.2 Astra and Sol: Cheaper, Shorter Answers, and the Fine Print
0dev.to