AI & ML
Speed limits for information acquisition in stochastic learning dynamics
arXiv stat.ML0 views
arXiv:2609.08219v2 Announce Type: replace-cross Abstract: Neural networks acquire internal representations through learning. In this work, we formulate stochastic gradient descent (SGD) as a Markovian stochastic process and derive a Fisher-information flow speed limit that bounds the rate at which trainable parameters can acquire information about data-generating parameters. The resulting inequality decomposes the upper bound on the information flow into drift and noise contributions, thereby quantifying the roles of deterministic learning forces and SGD-induced fluctuations from an information-theoretic perspective. We verify the bound in analytically tractable basis-function linear regression, where the information budget predicted by the bound reproduces the ordering and characteristic time scales with which different data-generating parameters are encoded in the learned parameters. These results establish Fisher-information speed limits as a quantitative framework for understanding which properties of the data-generating process are acquired, in what order, and how rapidly they can be encoded in the trainable parameters.
Source:https://arxiv.org/abs/2609.08219PreviousDeep Barycentric Regression for Optimal Transport Map Estimation and Its Statistical OptimalityNextSymplectic Hamiltonian Direct Discontinuous Galerkin Method for Wave Propagation
Related posts
Looking for first-time contributors: Open Catalog of AI System Prompts
0dev.to GitHub
How do you reduce LLM spam in PR reviews?
0dev.to GitHub

GH Profile Cyber Punk City Spin
0dev.to GitHub
Lost Your GitHub Repository? How to Privately Investigate and Safeguard Development Quality
0dev.to GitHub
FCC router ban: Netgear, Eero and Asus win Conditional Approval, TP-Link has not
0dev.to News
GPT Image 2.5 or Nano Banana 2.1? Cost per usable image
0dev.to Design