I wrote a monitoring stack (Prometheus, Grafana, Alertmanager) and a README that says "it just works". It had always worked on my own server. But I had set that one up by hand, over time, from memory. I had never once followed my own README from the first line to the last.
So I took a clean VM and did exactly that, the way a newcomer would. First on Ubuntu 22.04, then on 24.04 and 26.04. It took about a day.
I found 24 problems. Three of them stuck with me, because I'm the one who got them wrong.
The alert that fired on a healthy machine
On Ubuntu 22.04 a fresh VM with nothing running went to pending on a critical alert: FilesystemReadonly. My first thought was that I'd broken something.
It was my rule. systemd 249 mounts a read-only ramfs for its own needs, and my alert counted it as a failing disk. Ubuntu 24.04 and 26.04 don't do this, so on newer systems I'd never have noticed.
The fix is one word: add ramfs to the exclusion list. The more interesting part is that my own checks had missed it. I was verifying that targets are up and dashboards load. I wasn't verifying that a healthy system is quiet. Now /api/v1/alerts must be empty on every fresh stand.
The restore that restores nothing
My backup doc explained how to make a backup. Restore was one sentence without commands. So I tried it.
The Grafana archive has a top-level grafana/ directory, the Prometheus one doesn't. Unpack Grafana to the wrong path and nothing fails. You keep the old data and believe you restored.
I wrote the steps down and ran them. To be sure it really restored something, I created a folder with a marker in Grafana before the backup. After the restore it was there, and all 7 targets were up.
A backup you haven't restored is a hope, not a backup.
"Ansible 2.9+"
The README said Ansible 2.9 or newer. I had never run it on 2.9, I just wrote the number.
I tried. On 2.9 the syntax check passes, then the run fails on the target with ModuleNotFoundError, a known clash with Python 3.12. From 2.12 to the latest 2.21 everything worked, and the second run gave changed=0. The README now says 2.12+.
Everything else
The other 21 are smaller: a compose block in the wrong place, a stale link, flags in an example config that no code reads, Alertmanager needing restart instead of up -d. All fixed in 15 pull requests, #26 to #40.
What I did not check: a real x86_64 machine (I used emulation), Debian and RHEL, load.
A check I ran by hand once is a snapshot. The next version bump will break something again, so the next step is a CI smoke test: compose up, targets up, Ansible twice with changed=0. Docs should run like code.
How do you check that your install docs still work: CI, a manual run before a release, or you find out from an issue?
Repo: github.com/Airat71/monitoring-stack
Disclosure: I used an AI assistant to run the repetitive checks and to help draft this text. The findings, the decisions and the fixes are mine.
