In September we moved invoice reminders from an old CronJob in our billing namespace to the new billing service, which sends them from its own scheduler. The release was two pull requests: one turned the new reminders on, and one deleted the CronJob's manifest from the billing repository. Both merged on the same afternoon. Argo CD synced, the new reminders started, and we closed the ticket.
Six weeks later a customer forwarded us two reminders for the same invoice, sent on the same morning eleven minutes apart and worded slightly differently. One came from the new service. The other came from a CronJob we had deleted.
The billing Application had automated sync turned on, which we had read as Argo CD making the cluster match Git. It does, for everything that is in Git. Automated sync does not delete a resource that has disappeared from Git unless prune is also turned on, and prune is off by default, as a safety measure, so that a mistake in a repository cannot quietly delete production resources. The CronJob was still in the cluster, still on its schedule, sending reminders every weekday at eight.
Argo CD had told us. The Application had been OutOfSync since the afternoon of the release, with one resource marked as requiring pruning. The badge had been yellow for so long that people had stopped seeing it, and several of our other Applications were yellow too, for harmless differences nobody had cleaned up.
We deleted the CronJob by hand that morning and wrote to every customer who had received both reminders. Then we went through every Application and found seven more resources waiting to be pruned, including the Deployment and Ingress of a promotions page we had retired in the spring, still serving it.
Prune is now on for every Application, with the Prune=false sync option set on the few resources, such as persistent volume claims, that we never want removed automatically. Any Application that stays OutOfSync for more than an hour alerts its owning team, and we fixed the harmless differences so that a yellow badge means something again. Retiring a component now has its own checklist step: confirm in the cluster, not in Git, that it is gone.
We had treated deleting a file as the release. With prune off, it was a request waiting for someone to confirm it.
– Sergey Shinder
