AI & ML
Exact Tail Laws for Nonlinear SGD with Infinite-Variance Gradients
arXiv stat.TH0 views
arXiv:2610.23750v1 Announce Type: new Abstract: With infinite-variance gradient noise, SGD can have infinite expected excess objective even when most iterates are close to the optimum. We characterize its accuracy through exact first-order tail probabilities for the last iterate of nonlinear, constant-stepsize SGD. For regularly varying innovations of index $t\in(1,2)$, the leading measure transports one large gradient shock through the local Hessian flow. This is the same measure that governs the central stable Ornstein--Uhlenbeck limit; we prove its relative accuracy at shrinking tolerances above the central scale, with state-dependent noise and a growing observation horizon. The admissible window is nonempty for every available gradient moment $p>1$. The main technical step is a moment bound at the central fluctuation scale and a lower-moment control of the nonlinear error. The result yields explicit projection and excess-objective tails, extreme quantiles, and a factor $b^{1-t}$ for fixed batch size $b$. For affine population gradients, a sharper comparison gives the same law at fixed tolerances, including random sample Hessians. Coupled simulations in dimensions one to one hundred examine relative calibration, objective quantiles, batch effects, and the error due to random curvature.
Source:https://arxiv.org/abs/2610.23750PreviousWeighted spatial averages with missing observations: Convergence and accuracy of delta-method variance approximationsNextRobust Local Fr\'echet Regression: Nonasymptotic Theory and Minimax Optimality
Related posts

The Download: the agentic web arrives and the drive for safer roads
0MIT Technology Review

Stop counting tokens, start counting what ships
0MIT Technology Review
OpenAI is finally merging ChatGPT’s chat and Work tabs
0TechCrunch

10 Open-Source Robotics Repositories for 3D Perception, VLA Policies, and Sensor Calibration
0dev.to
My self-modifying system tried to wire a gate-passing module into its own protection enforcer. It was refused three times.
0dev.to
Last night an AI agent performed brain surgery on a mouse
0Hacker News