Backend
What Does the Rank Buy? A Spectral and Distributional Analysis of Low-Rank Adaptation
arXiv stat.ML0 views
arXiv:2609.32002v2 Announce Type: replace-cross Abstract: The rank $r$ in LoRA is widely treated as a capacity control: a smaller rank is assumed to yield a simpler model that generalizes better. We show that, under hard per-factor norm budgets---the idealization of the weight decay and norm control used in practice---this intuition breaks down. The reason is structural: under such budgets, the updates LoRA can reach are exactly the matrices of rank at most $r$ inside a nuclear-norm ball, and every complexity and displacement functional we analyze is maximized over this set by a rank-one update---so the rank cap never binds. The consequences follow directly. The linear-readout model class we study is identical for every $r \ge 1$, its Rademacher complexity carries no dependence on $r$, and the distance the adaptation can move the source distribution obeys a rank-independent upper bound that we show is sharp. If rank does not control capacity, where does it act? We identify two places. Statistically, replacing the per-factor budgets with a joint budget on the product restores a data-dependent, rank-sensitive complexity bound---though the gain appears only for well-spread feature distributions, and the worst case remains rank-free. Spectrally, rank sets the price of adaptation: canceling the leading singular directions of the pretrained weight requires both sufficient rank and sufficient budget. We bound the smallest rank achieving a desired source--target alignment, with upper and lower bounds that match under two-sided spectral decay. Together, these results recast rank as governing which updates are reachable and what cancellation costs---not how much capacity the model has.
Source:https://arxiv.org/abs/2609.32002PreviousService-Induced Congestion in Memory-Constrained LLM ServingNextFERPO: Forward Entropy-Regularized Policy Optimization
Related posts

Dr Amr Eloufy: How to Save Our Hair in This Fast-Moving World
0dev.to Watercooler
Only 6 of 45 Odds APIs Cover All 8 Big US Sportsbooks
0dev.to Data
Hadoop Ecosystem Overview
0dev.to Data
From One Chargeback to a Fraud Ring: Building Link Analysis That Works
0dev.to Data

Choosing the right AI web scraper depends on your needs, budget, and technical skills. This guide makes comparing the top tools easier and helps you find the right fit for your next project.
0dev.to Discuss

Choosing the right AI web scraper depends on your needs, budget, and technical skills. This comparison makes it easier to understand which tools are worth trying, from no-code options to developer-foc
0dev.to Discuss