AI & ML
Spectrum-Adaptive Generalization Bounds for Trained Deep Transformers
arXiv stat.ML0 views
arXiv:2605.07297v2 Announce Type: replace Abstract: Understanding why trained Transformers generalize well is a fundamental problem in modern machine learning theory, and complexity-based generalization bounds provide a principled way to study this question. While existing norm-based bounds for Transformers remove the explicit polynomial dependence on the hidden dimension, they typically impose fixed norm constraints specified a priori and can exhibit unfavorable exponential dependence on depth. In this paper, we derive spectrum-adaptive post hoc generalization bounds for multi-layer Transformers. Under layerwise spectral norm control, the bounds are expressed in terms of layerwise Schatten quantities of the query-key, value, and feedforward weight matrices. Since the Schatten indices need not be fixed a priori and can instead be selected after training, separately for each matrix type and layer, the bounds adaptively trade off spectral complexity against the dimension- and depth-dependent factors according to the learned singular-value profiles. Empirical comparisons of BERT-adapted proxies for the leading complexity factors suggest that the proxies induced by our bounds grow more slowly with depth and hidden dimension than the corresponding norm-based proxies. Overall, our results provide a complexity-based perspective on how the spectral structure of trained Transformers is reflected in generalization analyses.
Source:https://arxiv.org/abs/2605.07297PreviousSparse Network Inference under Imperfect Detection and its Application to Ecological NetworksNextMeasuring and Decomposing Mode Separation via the Canonical Diffusion
Related posts
P1.95 or P2.6: the pitch your camera picks before your client does
0dev.to
Building a Production AI Chatbot in Next.js: Architecture and Pitfalls
0dev.to
Home Assistant vs construir tu propia domótica en la nube: lo que aprendí haciendo la segunda
0dev.to
Hours of YouTube and 85 handtyped pages of notes on LLM Inference – Start here
0Hacker News
Humans will never again create a project like Wikipedia
0Hacker News
A Straight Line Is Not a Camera Pose: Auditing Short-Clip Pose Benchmarks
0arXiv cs.RO