Open Source
Why Kubernetes Cannot Contain a Full Lazy-Pulling Cache: Two Impossibility Results and a Bounded, Honest Design for Model Serving
arXiv cs.DC0 views
arXiv:2611.11739v2 Announce Type: replace Abstract: Lazy container-image pulling makes cold starts of autoscaled model-serving fleets on Kubernetes nearly independent of model size, but it moves the model's bytes into a node-level cache that the platform neither bounds nor accounts for. This article establishes, by experiment, two impossibility results for that cache. A monitoring-side remedy cannot catch its failure: a sentinel readiness probe stays green while 272 of 280 files are unreadable. A platform-side remedy cannot contain it: kubelet, given full disk-pressure visibility, frees 0 bytes by image garbage collection and falls back to evicting pods (26 across the campaign, the reading pod among them), because its reclamation actuator is keyed to an accounting that does not know the cache exists. The article then eliminates the failure class inside the component that owns the bytes, with no change to kubelet or the container runtime interface, by two design rules: a byte budget whose pressure signal is keyed to filesystem truth, so that a never-blocking, lossy accounting still enforces a hard bound (held against about 301,000 dropped updates), and pass-through semantics under which a failed cache write never becomes a failed read, with the coupling traced to a single io.MultiWriter. Across six pre-specified campaigns on AWS, upstream stargz-snapshotter fails 273 of 280 model files at 90% cache pre-fill; the proposed system reads all 150.3 GB with zero errors at +18.5% completion cost and holds the budget under a live read, co-tenancy, and a three-node fleet. A source audit shows the missing bound generalizes to SOCI and Nydus. The implementation is open source and has been proposed upstream.
Source:https://arxiv.org/abs/2611.11739PreviousAInfer-PD: Communication-Safe In-Place Prefill-Decode Multiplexing for Distributed MoE RolloutsNextIA-MoE: An Intensity-Aware Mixture of Experts for DNN Inference
Related posts
Turn a folder of MP4s into an editable publishing CSV with Python
0dev.to Python
Oriko CLI – Create your CV in the terminal with builtin Agent support
0Hacker News New
From 181 bytes: nixpkgs' full-source bootstrap
0Hacker News New
Show HN: Fufu – an opinionated Git client inspired by Jujutsu
0Hacker News New
Show HN: Pbc – a CLI for any PocketBase instance
0Hacker News Show

A Link You Can Just Open
0dev.to