Open Source
Why Kubernetes Cannot Contain a Full Lazy-Pulling Cache: Two Impossibility Results and a Bounded, Honest Design for Model Serving
arXiv cs.DC0 views
arXiv:2611.11739v2 Announce Type: replace Abstract: Lazy container-image pulling makes cold starts of autoscaled model-serving fleets on Kubernetes nearly independent of model size, but it moves the model's bytes into a node-level cache that the platform neither bounds nor accounts for. This article establishes, by experiment, two impossibility results for that cache. A monitoring-side remedy cannot catch its failure: a sentinel readiness probe stays green while 272 of 280 files are unreadable. A platform-side remedy cannot contain it: kubelet, given full disk-pressure visibility, frees 0 bytes by image garbage collection and falls back to evicting pods (26 across the campaign, the reading pod among them), because its reclamation actuator is keyed to an accounting that does not know the cache exists. The article then eliminates the failure class inside the component that owns the bytes, with no change to kubelet or the container runtime interface, by two design rules: a byte budget whose pressure signal is keyed to filesystem truth, so that a never-blocking, lossy accounting still enforces a hard bound (held against about 301,000 dropped updates), and pass-through semantics under which a failed cache write never becomes a failed read, with the coupling traced to a single io.MultiWriter. Across six pre-specified campaigns on AWS, upstream stargz-snapshotter fails 273 of 280 model files at 90% cache pre-fill; the proposed system reads all 150.3 GB with zero errors at +18.5% completion cost and holds the budget under a live read, co-tenancy, and a three-node fleet. A source audit shows the missing bound generalizes to SOCI and Nydus. The implementation is open source and has been proposed upstream.
Source:https://arxiv.org/abs/2611.11739PreviousAInfer-PD: Communication-Safe In-Place Prefill-Decode Multiplexing for Distributed MoE RolloutsNextIA-MoE: An Intensity-Aware Mixture of Experts for DNN Inference
Related posts
Birdsong's "language-like" Zipf law is matched by simple syllable rules, and song with no words passes the whale controls
0dev.to

AGPL vs MIT vs Open Core: Choosing Open Source Collaboration Software
0dev.to OpenSource
From exposed secrets to managed credentials: Reducing credential risk across the application lifecycle
0HashiCorp Blog
AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation
0arXiv cs.SE
Approximation of the L\'evy-driven stochastic heat equation on the sphere
0arXiv cs.NA
EgoSwarm: Learning Decentralized Robot Swarm Intelligence from Human Swarm Demonstrations
0arXiv cs.RO