Our images carried an empty commit label for five months because the ARG sat above FROM

Our pipeline builds every service image with docker build --build-arg GIT_SHA set to the commit being built. The Dockerfile turns that into a label, org.opencontainers.image.revision, and into an environment variable that the service prints at /version. In May someone tidied our shared Dockerfile template. The base image tag became an argument too, and all the ARG lines were gathered at the top of the file, above FROM, where they looked neat.

In October an order export started writing dates in the wrong format, and the first question in the incident channel was which commit was running. docker inspect showed the revision label as an empty string. /version returned an empty string. So did every image we had built since May, across nineteen services. We worked the commit out from deploy timestamps, which took forty minutes we did not have.

An ARG declared before the first FROM belongs to the global scope of the Dockerfile, and the only lines that can use it there are FROM lines. That is why our base image argument still worked. Inside a build stage the argument does not exist until the stage declares it again with a bare ARG GIT_SHA. Without that, LABEL and ENV expanded $GIT_SHA to nothing. Stages in a multi stage build work the same way: an argument declared in the builder stage is not visible in the final one.

Nothing complained. The argument had been declared, the build was green, the image ran, and the label was there with nothing in it.

We redeclared ARG GIT_SHA in each stage that uses it and fixed the template, which was the real source, so the next service copied from it would not start out broken. The pipeline now inspects every image it pushes and fails if the revision label is not exactly the commit it built from. The deploy job runs the same comparison against /version once the new pods are ready, so the label cannot be right while the running process says something else. The template itself has a test build that asserts both.

An argument above FROM looks as if it applies to the whole file. It applies to the FROM lines and nothing else, and a label with an empty value fails no check you have not written.

– Sergey Shinder