What happened
Our nightly auto-patching (a systemd timer running dnf upgrade) deliberately does not reboot. On days when the batch includes packages like the kernel or glibc — things that only take effect after a restart — it just logs REBOOT-NEEDED and leaves the reboot to a human.
The 33 piled-up updates from my previous post included exactly that: kernel and glibc. Rebooting the production site follows a simple runbook: put up the maintenance page → reboot → verify → take the page down. The server itself is down for under a minute. Yet the maintenance page was actually visible for about 10 minutes.
The slow part wasn't the server. It was me.
Procedure v1 failed — maintenance ON came too early
My original procedure looked like this:
# v1: maintenance ON → update → reboot sudo touch /var/www/html/.maintenance # maintenance page ON (flag file) sudo dnf upgrade -y --releasever=latest # ← takes 1–2 minutes here sudo reboot Downloading and applying packages takes a minute or two, during which the server can serve traffic just fine — but visitors see a maintenance page. Pointless downtime from the user's perspective. So I swapped the order:
# v2: update → maintenance ON → reboot sudo dnf upgrade -y --releasever=latest sudo touch /var/www/html/.maintenance sudo reboot Now the only outage window is around the reboot itself — under a minute. In theory.
Measured: 10 minutes — the bottleneck was human
| Step | Expected | Measured |
|---|---|---|
| dnf download/apply (before maintenance) | 1–2 min | ~2 min (no impact) |
| Maintenance ON → reboot → OS back up | ~1 min | ~50 s |
| Human reopens browser SSH and logs back in | (not budgeted) | ~9 min |
Removal command (rm) | seconds | seconds |
Checking uptime -s afterwards showed the reboot itself finished in about 50 seconds. The remaining 9 minutes were me: the browser SSH session drops on reboot, I reopen the tab, retry a few connection errors wondering "is it up yet?", get distracted by something else, and finally log in and type the removal command.
I had designed the procedure to minimize server downtime — but I was measuring the wrong thing. What determines how long the maintenance page stays up isn't the reboot time; it's the time until a human types the next command.
The fix — take the human out of the removal step
Two changes.
(1) Prepare the one-liner to paste after reconnecting
Right before rebooting, put the next command in a note:
sudo rm -f /var/www/html/.maintenance && curl -sk -o /dev/null -w '%{http_code}\n' https://localhost/ Reopen the session, paste, done. No "wait, what do I type after logging in?" time.
(2) A systemd oneshot that clears the maintenance flag after reboot
# /etc/systemd/system/maintenance-clear.service [Unit] Description=Clear maintenance flag after reboot After=httpd.service Requires=httpd.service [Service] Type=oneshot # Wait until localhost returns 200 before clearing the flag ExecStartPre=/usr/bin/bash -c 'for i in $(seq 1 30); do curl -sfk -o /dev/null https://localhost/ && exit 0; sleep 2; done; exit 1' ExecStart=/usr/bin/rm -f /var/www/html/.maintenance [Install] WantedBy=multi-user.target The key is the ExecStartPre wait. If you remove the flag unconditionally at boot, the maintenance page disappears before the app is up and visitors see a 502. Verify "healthy means exactly this value" first, then remove. If the health check never passes, the flag stays and the maintenance page remains — the failure mode is safe.
With this in place the human part of the runbook ends at "dnf → maintenance ON → reboot". I only come back later to check uname -r.
Verification
What I checked after the reboot:
uptime -s # reboot time (backs up the actual outage duration) uname -r # running on the new kernel? sudo dnf history info last # what was applied On top of that: during maintenance, I confirmed the four main pages returned 200 from inside using a bypass cookie; after removal, I recorded 200s and response times for the same four pages via their external URLs. The recovery time I log is not "when I removed the flag" but when the site looked healthy from outside.
Takeaways
- Nightly auto-patching doesn't reboot by design. Kernel/glibc days need a human reboot
- Turn maintenance ON right before the reboot — showing it during the download is pointless exposure
- Measure downtime not as "time the server was down" but as "time until the human types the next command." Here: 50 seconds vs. 10 minutes
- Bundle the removal into the reconnect one-liner, or automate it with a health-checked oneshot, and the "human" row disappears from the timetable
Add a "human" row to the time estimates in your runbooks. It's usually the longest one.
