Our stock availability service keeps a snapshot of every depot's stock in memory. In September we stopped building that snapshot inside the service at startup and moved the work to an init container, which downloads the latest snapshot from object storage, unpacks it and indexes it onto a shared emptyDir before the service starts. Unpacking is memory hungry, so the init container got a memory request of six gibibytes. The service itself requests five hundred and twelve mebibytes and has never used much more.
The week after, the cluster autoscaler added nine nodes, and our cost dashboard noticed before we did. The nodes were real and paid for. Their memory use was under fifteen percent.
The scheduler does not add an init container's request to the pod's. It takes the larger of two numbers: the highest request among the init containers, and the sum of the requests of the app containers. That larger number is the pod's effective request. It is what the scheduler places the pod by, and it is held for the life of the pod. Our init container finished in forty seconds. Each of our twenty four replicas then reserved six gibibytes of node memory for days while using under one. A hundred and forty four gibibytes promised for about twelve in use, and the autoscaler bought nodes to cover the promise, because promises are all it reads.
The documentation states the rule plainly, in the section on init containers and resources. Nobody reviewing our pull request had connected a forty second step with the size of the whole pod for its whole life.
We moved the heavy work out of the pod. A job now runs once per snapshot, builds the index and writes it back to object storage, so the init container only downloads a finished file and requests two hundred and fifty six mebibytes. The nine nodes were gone within the hour. Our CI now renders every chart, computes each pod's effective request the way the scheduler does, and fails a pull request in which an init container asks for more than twice what the app containers ask for, unless the change explains why. Our capacity dashboard shows the effective request next to the declared one.
A container that runs for forty seconds can still decide what a pod costs for a week.
– Sergey Shinder
