AI & ML
Settling the Burn-in Gap: Sharp Risk Bounds for Tail-Averaged SGD in Linear Regression
arXiv stat.ML0 views
arXiv:2611.13383v1 Announce Type: cross Abstract: We study the excess risk of constant-stepsize online stochastic gradient descent (SGD) with tail-averaging for high-dimensional linear regression. Building on the instance-dependent spectral framework of Zou et al. (2021), we establish matching initialization-explicit upper and lower bounds on the total excess risk of tail-averaged SGD in terms of the population covariance spectrum $\mathbf{H}$ and initialization $\mathbf{w}_0-\mathbf{w}^*$ whenever a trajectory spectral feedback condition holds. Existing analyses exhibit a fundamental gap in the effective variance: the upper bound of Zou et al. (2021) retains an undamped contribution from the initial error in the head eigenspace $\mathbf{I}_{0:k^\dagger}$, whereas their lower bound omits the variance induced by the head initialization altogether. We show that both the existing upper and lower bounds are loose: the single-hop fast-mode variance contribution from the head initialization is exponentially damped during the burn-in phase, whereas fourth-moment energy scattered from the head initialization into slow spectral modes during the early burn-in steps survives undamped into the averaging window. We prove that our spectral feedback condition is satisfied in two concrete constant-stepsize regimes: (i) for arbitrary covariance spectra $\mathbf{H}$ and any finite horizon, under an explicit horizon-dependent stepsize chosen prior to execution; and (ii) for polynomially decaying spectra $\lambda_i\asymp i^{-a}$ ($a>1$), under a single horizon-independent positive stepsize. Finally, we complement these positive results with explicit one- and two-dimensional separation constructions proving an exponential-versus-polynomial gap in the lower bound of Zou et al. (2021) and showing why fourth-moment assumptions alone cannot determine the asymptotic exponential decay rate without additional structural conditions.
Source:https://arxiv.org/abs/2611.13383PreviousPrediction Inference for Hybrid Models on Realized Volatility ForecastingNextSufficient Dimension Reduction with Multimodal Predictor Objects
Related posts
EEG-X: Toward Device-Agnostic and Noise-Robust Foundation Model for EEG
0arXiv cs.HC
Minority Reports: Balancing Cost and Quality in Ground Truth Data Annotation
0arXiv cs.HC
Whose Guardrail? User Identity Shapes the Effectiveness of Model Disclaimers in Mitigating LLM Political Persuasion
0arXiv cs.HC
Art2Song: Enhancing Visual Art Appreciation with Contextual Music Generation
0arXiv cs.HC
Characterising divergence between heuristic and language-model ratings of mobile user interface quality
0arXiv cs.HC
Exploring Cross-Reality Transitions between Projections and Head-Mounted Displays for Immersive Digital Art
0arXiv cs.HC