AI & ML
Phase Transition from Random Features to Fully Trainable Networks
arXiv stat.ML0 views
arXiv:2610.31663v1 Announce Type: cross Abstract: Constructing accurate and computationally tractable approximation spaces is a fundamental task in scientific computing. Partially trainable neural networks bridge fixed random features and fully adaptive representations. This paper quantifies a phase transition pattern in approximation bounds and observes analogous behavior in training errors: a rapid but smooth initial improvement followed by diminishing returns, with a sharper transition under greater initialization mismatch. Under polynomial spectral decay with exponent $\alpha>1$ and leverage compatibility, we establish a squared $L_2$ approximation rate $\mathcal{O}((\kappa_P\log(m/\delta)/m)^\alpha)$ for two-layer networks of width $m$ and trainable proportion $P$, where $\kappa_P=(P/\kappa_1^*+(1-P)/\kappa_0^*)^{-1}$. The bound holds uniformly over the unit ball of a reproducing kernel Hilbert space (RKHS) with probability at least $1-\delta$. The constants $\kappa_0^*$ and $\kappa_1^*$ quantify the fixed and trainable groups' sampling mismatches relative to the target RKHS. For spherical biased-ReLU features in dimension $d$, we verify these assumptions with $\alpha=(d+2)/(d-1)$ and $\kappa_0^*=4\sqrt{2}\tau(d+1)/d$ for initialization scale $\tau\ge1$, and $\kappa_1^*=1$ under ridge leverage sampling. For $d=5$ and $\tau=10$, a trainable proportion of approximately $4.1\%$ captures $90\%$ of the reduction in the approximation bound from $P=0$ to $P=1$. A structured three-layer example illustrates how a few trainable first-layer neurons exploit fixed downstream blocks through nonlinear composition. Numerical experiments across diverse settings, including networks with up to 15 layers, exhibit similar phase transitions in training errors.
Source:https://arxiv.org/abs/2610.31663PreviousSympFormer: Inertial Attention Blocks from Damped Hamiltonian Dynamics on Spaces of Probability MeasuresNextEluder-Dimension-Free Regret for Stage-Wise Constrained Contextual Bandits via Offline Regression
Related posts
Introducing the Grid Protectors Program
0OpenAI Blog
The world has a new plan to save nature, but it still needs the money
0Ars Technica

Could data center backlash flip Florida?
0The Verge

Vivo’s first Pro Max phone proves its Ultra is still in a league of its own
0The Verge

Why Peacock is making a huge bet on YouTube
0The Verge
Ninja’s first-ever microwave just replaced two of my kitchen appliances at once
0TechCrunch