Intro
One morning my security-research routine fired an urgent alert: "PHP 8.3.37 (released 11/19) contains an important fix. Production is on 8.3.36 — not applied. Patch ASAP."
I rushed into the server. Production was already on 8.3.37, php-fpm restarted and all. The previous night's auto-update timer had finished the job. The box that was actually behind was dev. We even have a gate — "dev goes first, production only runs if dev succeeded" — so how did production overtake dev? The cause was a one-hour gap between the repository publishing the package and the two timers firing.
The setup
Dev and production run the same Linux distribution, and a systemd timer applies security updates every night:
- 23:00 — dev's timer runs
dnf upgradeand records the result in a state file - 00:00 — production's timer runs
dnf upgradeonly if dev's state file says "success", and restarts php-fpm if PHP was updated
Separately, a web-only research routine reads vulnerability feeds every morning and reports. It never logs into the servers.
Gotcha 1: the alert's "not applied" never looked at the server
The research routine bases its report on exactly two facts: "8.3.37 was published on 11/19" and "production was 8.3.36 at the last check." It doesn't see the server's current state, so if the patch lands between publication and the next check, it will always shout "not applied."
The alert isn't wrong. The work was simply finished somewhere the alert can't see. Even when it says urgent, look at the actual server first — this time that took ten minutes and settled everything.
Gotcha 2: a "dev first" gate can't stop production from overtaking dev
Put the timeline side by side:
timeline title What happened on the night of 11/19 23:00 : dev timer runs : 8.3.37 not in the repo yet : records "success" with 0 updates 23:xx : repo publishes 8.3.37 00:00 : production timer runs : dev state file says "success" : applies 8.3.37, restarts php-fpm The gate only checks dnf's exit code — "did the run succeed?" It never checks "is production about to apply the same packages dev did?" So when the repo publication falls between the two timers, dev records "success with 0 updates," production records "success with the new version," and production ends up newer than dev. The gate was never violated once, yet the ordering it was built to protect didn't hold.
Verification: reconstructing "when was it applied" after the fact
Three commands pin down when production got the patch:
# package install time rpm -q php-fpm --qf '%{VERSION}-%{RELEASE} %{INSTALLTIME:date}\n' # when php-fpm last (re)started systemctl show php-fpm -p ActiveEnterTimestamp # dnf transaction history (time and packages) dnf history list php\* dnf history info <ID> All three timestamps agreed on just-past-midnight, matching the timer's journal log. I manually ran dnf upgrade on dev to bring it to 8.3.37, then confirmed zero pending updates on both:
dnf check-update --security | grep -c . || echo "0 pending" The fix: put "what" into the gate, not just "whether"
The proper fix is to have dev's timer record the list of packages it applied in the state file, and have production apply only that list:
# dev: record what was applied dnf -y upgrade --security | tee /var/lib/update-gate/applied.txt # production: apply only what dev applied (sketch) xargs -a applied-packages.txt dnf -y upgrade I haven't implemented it yet. The trigger condition is a narrow window — "repo publishes between 23:00 and 00:00" — and the actual harm was "production became safe one hour early." I wrote down that if the ordering flips again, I'll build it. Match the fix to the size of what actually happened.
Takeaways
- A web-only alert answers "what came out in the world." A server check answers "what are we missing." Different questions — even an urgent alert means look at the server first
- A "dev first" gate only sees success/failure. Whether the same content was applied is checked by no one unless you put it in the gate
- If the repo publishes between the two timers, production overtakes dev. The time gap between timers is exactly your race window
- Apply times can be reconstructed later with
rpm -q --qf '%{INSTALLTIME:date}',systemctl show -p ActiveEnterTimestamp, anddnf history
The better your automation runs, the more the human's job shifts toward doubting the alerts. The alert doesn't lie — the work is just already done somewhere it can't see. This was the most harmless version of that.

