AI & ML
Decomposing SGD Dynamics in Neural Networks: Teacher-Induced Spikes and Variance Inflation
arXiv stat.ML0 views
arXiv:2611.24908v1 Announce Type: new Abstract: We study two-layer neural networks trained by stochastic gradient descent (SGD) on a multi-index teacher model, focusing on how first-layer parameters orthogonal to the teacher subspace affect generalization. We decompose the first-layer weights into a signal component aligned with the teacher and an orthogonal complement, and analyze their joint SGD dynamics together with the second layer in the proportional limit. We show that although SGD updates to the complement are initially isotropic, the training dynamics induce teacher-dependent spectral spikes in the complement Gram matrix. This emergent anisotropy provides a concrete mechanism by which representation variance is amplified, and generalization degrades. To isolate this effect, we introduce Decomposed Dynamics, an analytically tractable surrogate whose asymptotic test-error dynamics coincide with those of SGD. In simulations with a ReLU student, freezing the directions of the orthogonal complement improves generalization relative to training them. Finally, we show that the signal component rapidly aligns with the second-layer weights, revealing strong cross-layer coupling even in this shallow setting. Together, our results provide a dynamical explanation for how feature learning outside the teacher subspace harms generalization.
Source:https://arxiv.org/abs/2611.24908PreviousA Generative Model for Higher-Order Network DynamicsNextExploration via Exploitation: The Blessing of Reward Diversity in Personalized Federated RL
Related posts
Mesa 26.3.1 Ships Fixes For Intel Jay Compiler & RADV Driver
0Phoronix

Report finds workers are staying around at their jobs — but many say that's because they feel stuck
0TechRadar

95+ best Black Friday deals 2026 in the UK — GTA 6, Shokz OpenRun, Shark vacuums, Oura Ring, Kindle, Switch 2, and more
0TechRadar
Gig Work Is Invading Nursing Homes. The Risks Can Be Deadly
0Wired
4 Request-Validation Gotchas When Migrating OpenAI Clients to GPT-6.2 Reasoning Models
0dev.to
Razorpay webhook not verified: anyone can fake a payment
0dev.to