AI & ML
Factual recall in linear associative memories: sharp asymptotics and mechanistic insights
arXiv stat.ML0 views
arXiv:2605.10795v2 Announce Type: replace Abstract: Large language models excel at factual recall, yet the fundamental limits of storing input--output associations with neural networks remain unclear. We study these limits in a minimal setting: a linear associative memory that maps $p$ input embeddings in $\mathbb{R}^d$ to their $d$-dimensional targets via a linear layer, requiring each mapped input to score its own target above all other outputs. Since all inputs compete for the same outputs, these constraints are strongly correlated, which makes the storage capacity $p_c$ --- the largest number of associations that can be stored --- hard to characterise. We address this in three steps. First, we introduce a decoupled model, in which each input competes with its own independent set of outputs, and provide numerical and analytical evidence that it is equivalent to the original model, both in capacity and in the structure of the learnt solution. Second, using tools from statistical physics, we show that the decoupled model has capacity $p_c \log p_c / d^2 = 1/2$, extend this computation to linear two-layer networks, and go beyond the capacity by predicting the spectrum of the learnt weights and the storage mechanism. The latter explains how the optimal solution outperforms a na\"ive Hebbian rule: rather than boosting input--output alignments with broad fluctuations, it raises the correct scores just above the extreme-value threshold set by the competing outputs. Third, we prove the decoupled model's capacity thresholds rigorously, including in the non-convex two-layer case; combined with a recent proof for the original model, this shows that both models have the same capacity for full-rank weights. These findings give a sharp characterisation of factual storage in linear networks, and suggest decoupling as a general tool for memorisation and learning problems in which each input must be matched to one of an extensive number of outputs.
Source:https://arxiv.org/abs/2605.10795PreviousEstimating Implicit Regularization in Deep LearningNextData-Driven Stochastic LQR Control with Probabilistic Guarantees and Uncertainty Refinement
Related posts
Meta Quest Promo Codes and Coupons for November 2026
0Wired
BullMQ in NestJS: Retries, Idempotency and Dead-Letter Queues
0dev.to
80.6% at Chores, 12% at Real Work: AI's Benchmark Week
0dev.to
Efficient estimation of regression models for recurrent events in presence of terminal events
0arXiv stat.CO
Local Whittle estimation for distributional time series: A regularization framework in Bayes Hilbert space
0arXiv stat.CO
Inventory and Assortment Optimization under General Choice: A Unified Approximation Framework
0arXiv stat.CO