Our label workers take print jobs off RabbitMQ and send them to the printers in our depots. For three years they ran on Docker Swarm, and in September we moved them to Kubernetes with the rest of our services. On a Saturday night in October the RabbitMQ cluster failed over to another node during maintenance. Four of the six workers reconnected. Two stayed connected to nothing, and from midnight until a quarter past nine two depots printed a third fewer labels than they should have, while the jobs waited in the queue for the four that were still working.
The workers have a known weakness. After some broker failovers the client library does not notice that its connection is gone, and the consumer loop sits waiting for messages that will never arrive. We had dealt with it years ago. The loop touches a heartbeat file after every message and every idle poll, and the image has a HEALTHCHECK that fails when the file is more than two minutes old. On Swarm a task whose health check fails is replaced, so a wedged worker was gone within a few minutes and nobody was ever paged.
Kubernetes does not run the HEALTHCHECK in an image. It is part of the image configuration Docker reads, the kubelet ignores it, and a pod is restarted only for its own liveness probe, defined in the pod spec. Our worker pods had no probes at all. Kubernetes saw a running process and kept it running for nine hours.
The migration checklist had translated everything in our stack files, the replicas, the resources, the restart policy and the networks. The health check was not in the stack file. It was in the Dockerfile, and docker inspect on the image still showed it, which made it look as if it was still there.
The worker now has a liveness probe that runs the same command against the same heartbeat file, with a period of thirty seconds and a failure threshold of four. It checks only the worker's own loop and nothing downstream, so a slow broker or a slow printer does not restart every pod at once. We also searched every Dockerfile we own for HEALTHCHECK and found three more images whose checks had stopped doing anything when they moved.
The check had come with us into the new cluster, inside every image. Kubernetes never ran it, because it was written in the format of the platform we had left.
– Sergey Shinder
