AI & ML
Scaling Limits of Long-Context Transformers
arXiv stat.TH0 views
arXiv:2605.08505v2 Announce Type: replace-cross Abstract: We study the long-context limit of softmax self-attention with a fixed query and a random context of $n$ i.i.d. keys on the sphere with the intrinsic dimension of the key distribution held fixed, viewing the inverse temperature $\beta_n$ as the scaling parameter that decides whether attention degenerates into uniform averaging or collapses onto the single closest key. We show that the critical scale at which selectivity emerges is determined by the local exponent of the distance-to-query distribution near zero rather than by global features of the context, and scales like $\beta_n^\ast \asymp n^{2/(d-1)}$ for uniform keys on $\mathbb{S}^{d-1}$. Furthermore, we characterize the limiting laws of the ordered attention weights and of the attention output across all regimes of $\beta_n$: a subcritical regime in which the output reduces to a local average around $q$ with explicit deterministic bias and Gaussian fluctuations; a critical regime in which a finite collection of nearest keys retains macroscopic mass without single-key collapse; and a supercritical regime in which all mass concentrates on the closest key. Of notable interest is the subcritical case with identity value matrix where the attention map approximately implements a backward heat equation.
Source:https://arxiv.org/abs/2605.08505PreviousMAGDrop: Momentum Adaptive Gradient Dropout With Provable Generalization GuaranteesNextThe Symmetric Location Problem: a Song of Efficiency and Robustness
Related posts
Global Test of Functional Effects in Mean Field Variational Bayesian Scalar-on-Function Regression
0arXiv stat.CO
ASUS’ New ProArt P16 Is Built for AI Creators, With a 20-Core CPU and 6,144-Core GPU
0FOSS Bytes
Rails Versions 8.0.5.2 and 8.1.4.1 have been released!
0Ruby on Rails
Ancient Egypt set taxes by a gauge on the Nile. What is the one reading your system depends on?
0dev.to Data
Which AI chatbot does each industry depend on? Referral shares for 2,429 top websites (September 2026)
0dev.to Data
Benefits of Installing Vinyl Flooring in Residential Spaces
0dev.to Design