Our rollout finished in two minutes and the new pods started crashing in the fifth

In May release 7.3 of our delivery tracking API went out at eleven in the morning. The pipeline watched kubectl rollout status, which reported success after two minutes, and marked the deploy green. By the seventh minute every one of the twelve new pods had restarted at least once, and none of the old ones was left to take the traffic.

The release added a background task that pushes delivery events to a reporting database, first five minutes after start and every five minutes after that. Its credentials existed in staging and not in production. The service is Node, the first push failed with a rejected promise that nothing caught, and since Node 15 an unhandled rejection ends the process by default. So each new pod started, passed its readiness probe, served requests normally for five minutes and died. After a restart it did the same, and soon they were all in CrashLoopBackOff.

A rolling update moves forward on available pods. A pod is available once it has been ready for minReadySeconds, and our Deployment had never set it, so the default of zero applied. Each new pod counted the moment its readiness probe passed, the controller scaled the old ReplicaSet down to match, and twelve new pods had replaced twelve old ones well before any of them reached its fifth minute. Our rollout was designed to stop if new pods failed, and they failed only after it was over.

The alert on error rate fired at eight minutes and kubectl rollout undo brought the previous version back in about ninety seconds. The fix to the code was a missing secret and a catch around the task.

Background tasks now run once at startup, before the readiness probe can pass, so a bad credential fails the pod before it is ever counted. The Deployment sets minReadySeconds to sixty to catch what that misses, and progressDeadlineSeconds makes the rollout fail if pods never stay up that long. The pipeline no longer stops at rollout status. It watches restart counts and the error rate for fifteen minutes afterwards and rolls back by itself if either rises. And every secret a release reads is listed in its manifest and checked against the target environment before the deploy starts.

Ready told us a pod could answer at that second. We had let it stand for whether the release would last.

– Sergey Shinder