We package our depot agent as a .deb and publish it to our own apt repository. Fifty depot servers, all Ubuntu, run apt-get update and then install the agent every night, and the job reports success to our monitoring when both commands exit zero.

The repository's Release file is signed with a GPG key we created three years ago with a three year expiry. Someone diligent noticed in the summer that it was due to expire in October and extended it with gpg --quick-set-expire on the signing host. Publishing kept working through the expiry date, and so did the nightly job.

Five weeks later a depot reported a bug we had fixed in October. The server was still on the September release. So were the other forty nine, and we had published three releases since.

The extension had happened in one place. Each depot had its own copy of our public key, in the keyring file the image had put in /etc/apt/keyrings three years before, and that copy still carried the old expiry date. On the day it passed, apt on every depot saw a valid signature from a key it believed had expired, EXPKEYSIG, and printed a warning: the repository is not updated and the previous index files will be used. A warning, with exit code zero. The install that followed looked at the old index, found the agent already at the newest version it knew about, and did nothing, successfully.

We pushed the refreshed public key to every depot by hand that morning, and the next run brought them all up to date.

The key now travels through apt itself. A small package, depot-archive-keyring, installs the keyring file, and when we extend or rotate the key we publish the new keyring package weeks before the old expiry, so depots pick it up while they can still verify it. The nightly job runs apt-get update with --error-on=any, which makes these warnings fail the command. After the install, the job compares the agent version it has with the newest version in our repository and reports a failure if they differ. A calendar alert fires ninety days before the signing key expires, and the runbook for it starts with the clients.

A signing key has a copy at every machine that trusts it. We had renewed the one we could see.

– Sergey Shinder