We release our pricing service one region at a time. The pipeline runs helm upgrade with --wait in the first region, and when Helm reports success it moves on to the next, on the reasoning that a release healthy in one place is safe for the others. In October a release reached all three regions in twenty minutes, and in all three our quotes went out with the previous year's fuel surcharge.
The chart contains a Job that loads the current tariff tables into the service's database after each upgrade. It is an ordinary resource, not a Helm hook, and its name carries the release revision, because a Job's pod template cannot be changed once the Job exists and Helm would otherwise fail trying to update it. In this release the job could not read the bucket holding the tariff files, because a bucket policy change had gone out the day before. It failed, retried and kept failing.
Helm said success anyway. In Helm 3, --wait waits for the pods, claims and services in the release to be ready, and for Deployments and StatefulSets to reach their minimum number of ready pods. It does not wait for Jobs. That takes a second flag, --wait-for-jobs, which makes Helm wait for every Job in the release to complete within the same timeout. Our new pods were ready in two minutes, serving whatever tables the last successful load had left behind, and Helm marked the release deployed while the job was on its second retry.
The gate between regions checked the signals that could not see the failure. Quotes were still being returned, error rates were flat, and a stale surcharge is not an error on anybody's dashboard.
The pipeline now passes --wait-for-jobs, so a failing load fails the release in the first region and stops it there. The tariff job records the version of the tables it loaded, the service exposes that version, and the gate compares it with the version the release was meant to carry. The release script also checks that every Job the release created has succeeded before anything is promoted, because we would rather not depend on a single flag again.
Success from a deployment tool is a statement about the things it was told to watch. Ours was watching the pods and had never been asked about the job that mattered.
– Sergey Shinder
