AI & ML
Learning by Compression: Risk Estimates Attain the Law-of-Large-Numbers Rate
arXiv stat.ML0 views
arXiv:2610.13442v1 Announce Type: new Abstract: Compression schemes provide a general framework for deriving data-dependent risk bounds for learning algorithms. For learning rules based on preferent compression, existing results upper-bound the risk in terms of a sample-dependent complexity $\boldsymbol{k}$, defined as the cardinality of the compression augmented by the training examples on which the reconstructed hypothesis is inappropriate. These results, however, yield a margin of order $O(\sqrt{\ln N/N})$ above $\boldsymbol{k}/N$. In this paper, we establish that the logarithmic factor is not intrinsic. We prove that, for every fixed confidence level $1-\delta$, the risk is bounded with probability at least $1-\delta$ by $$ \frac{\boldsymbol{k}}{N} + O\left(\frac{1}{\sqrt{N}}\right), $$ uniformly over all possible values of $\boldsymbol{k}$. Thus, risk estimates for learning algorithms based on preferent compression attain the same law-of-large-numbers rate as the empirical estimate of the probability of a fixed event. We also show that the leading term $\boldsymbol{k}/N$ is necessary and investigate its relation to the empirical risk. In particular, we identify regimes in which the empirical risk retains the $O(1/\sqrt{N})$ convergence rate, as well as regimes in which it converges more slowly or ceases to be consistent, while the complexity-based estimator preserves the $O(1/\sqrt{N})$ rate.
Source:https://arxiv.org/abs/2610.13442PreviousModel-free Bayesian Clustering and Multimodal Posterior SummarisationNextDefence against the dark arts of distillation: Unsupervised discovery of subliminal learning
Related posts

Use it, or lose it 🧠
0dev.to AI

AI Daily Digest November 3, 2026: OpenAI Wires Agents into ServiceNow, Australia Books a 40,000-GPU AI Factory
0dev.to AI
How Publishers Can Test Engagement Signals to Improve On-Site Discovery
0dev.to AI
What Meta's Muse actually reads: first data from 3 tiny businesses
0dev.to WebDev
Your AI apps should answer to you, not to their lab
0dev.to Security

Warp's Breach Reached Agent Decision Logs. What One Agent Trace Actually Contains
0dev.to Security