AI & ML
When Thoughts Become Memory: Unifying In-Context Learning and Fine-Tuning
arXiv stat.ML0 views
arXiv:2610.29419v1 Announce Type: new Abstract: We are the first to give a certified rule for whether unseen data belong in a pretrained model's context or its weights, built on an exact law for their information value. That value, the code length saved on held-out queries, equals in expectation the bits stored about the query, plus a prior repair, minus a decoding loss, for any model, architecture, dataset and task. The law's ceilings certify the choice from one measurement, and an anytime-valid test makes it from held-out queries, with error controlled whenever it stops. Every certificate the ceilings issue on a trained transformer, TabPFN, an occupancy network and a stereo network is correct. The test picks TabPFN over XGBoost and a trained surrogate over an in-context learner on airfoil pressure, calls the better of GPT-2's cache and fine-tune in 61,948 of 64,000 query streams over 32 comparisons, and errs in at most 3.3% of null streams. The law also diagnoses when learning in context helps and when it does not. Deleting part of a context is held-out validation of that part in any model, and adding parts one at a time traces the learning curve in context: one patch that sees a shape feature cuts an occupancy network's code length there a hundredfold, while eight that miss it nearly double it, and one camera that sees the surface beats four in a stereo network. Ranked by this label-free valuation, a kriging learner's three best support cars beat four drawn at random, and on a yes-or-no claim the valuation recovers a published evidence adjudicator, with AUROC 0.91.
Source:https://arxiv.org/abs/2610.29419PreviousSchr\"odinger-Bass Bridge Barycenter Models for Measure-Valued DataNextA unified approach for estimation of unnormalized models via Stein operators
Related posts

Putin demands Russian data centers supply capacity to the state in wake of Ukraine drone attacks
0Tom's Hardware

OpenAI considers rebranding data centers to 'AI compute campuses" as data center backlash intensifies
0Tom's Hardware
What tech entrepreneurs can teach scientists about failure
0Nature News

X for iPhone and iPad just added over a dozen new app icon options
09to5Mac
iOS 27.2’s clever new Messages feature can be tailored to your needs
09to5Mac

OpenAI is retiring ChatGPT Classic app, will soon combine Chat and Work tabs in new app
09to5Mac