Our checkout Deployment had replicas: 10 in its manifest for two years, and our pipeline applies manifests with kubectl apply. In October we added a HorizontalPodAutoscaler with a minimum of ten and a maximum of forty ahead of Black Friday.
The Kubernetes documentation is clear that once an autoscaler owns a Deployment, the replicas field should come out of the manifest. Otherwise every apply sets the count back to the number in the file, whatever the autoscaler had decided. So a second pull request deleted one line, replicas: 10. It merged at twenty past twelve on a Wednesday. Two minutes later checkout was running on one pod.
kubectl apply decides what to change by comparing three things: the manifest you give it, the live object, and a copy of the previous manifest it keeps in an annotation on the object, last-applied-configuration. A field that was in the previous manifest and is missing from the new one is a field you deleted, so apply removes it from the live object. A Deployment without replicas gets the default, which is one. The Deployment controller began terminating nine pods. The autoscaler saw one pod against a minimum of ten and asked for ten again within a minute, but a new checkout pod takes about ninety seconds to pass its readiness probe, and for nearly three minutes one pod took the lunchtime traffic. Errors peaked at forty percent of requests.
The same page of the documentation warns about this a paragraph further down and says how to avoid it: take the field out of the stored annotation first, with kubectl apply edit-last-applied, so that apply no longer thinks it owns the field. We had read the first paragraph.
What changed is how we review a deploy. The pull request had shown one deleted line, which looked like the smallest change possible. The pipeline now runs kubectl diff against the cluster for every pull request and posts the result, and that output would have shown replicas going from 10 to 1. Any change that lowers or removes the replica count of a Deployment with an autoscaler needs a second approver. Our autoscaler checklist now puts the annotation step before the manifest change.
A line deleted from a manifest is not a line Kubernetes forgets. It is an instruction to put the default back, and the default was one.
– Sergey Shinder
