Researchers analyse two-layer networks trained by stochastic gradient descent on a multi-index teacher model. They decompose first-layer weights into teacher-aligned signal and an orthogonal complement, showing SGD induces teacher-dependent spectral spikes in the complement Gram matrix and affects generalisation through variance inflation.
Why it matters
It clarifies how supposedly irrelevant directions can harm or shape generalisation, informing training and regularisation choices.
Analysis
The work separates signal from orthogonal dynamics in a tractable proportional limit, revealing emergent spikes despite initially isotropic updates. That offers a precise mechanism for variance inflation. The main uncertainty is whether the effect persists in deeper, realistic networks and practical training regimes beyond the two-layer teacher setting.
Source: arXiv
AI disclosure: this article was produced with AI assistance and reviewed for accuracy.
