After the first article a reader (@devenquan) wrote that they had run the same exercise on their own tool and found two problems. Their suggestion: do it once more, on a machine that already has stuff on it. Fair point. I had never tested it that way, so I ran it.

I used an Ubuntu 24.04 VM (arm64) with Docker 29.9.2 and followed my install docs the way a newcomer would. Five scenarios: busy ports, an old Grafana volume with a different password, an upgrade over existing data, Compose and Ansible on one host, and a node_exporter from apt already holding port 9100.

Before the run I wrote down what I expected: the docs cover none of these. They covered none. Two of the five I staged myself.

My CI starts the same stack on a clean runner. There the exit codes and changed=0 are green and the stack is up. On this VM they were green too, and the stack was not. Green output alone does not tell you whether the stack is up.

"Healthy" with no network

I took ports 9090 and 3001 and ran docker compose up -d. It fails on the first port, three containers run, two stay in Created. Then I ran it again, because that is what a newcomer does. Prometheus showed Up (healthy) with no network and no published port. On the host, 127.0.0.1:9090 answered with the other process.

My healthcheck requests localhost from inside the container, so it can't fail. And plain docker compose ps hides containers in Created: the Quick Start told you to run exactly that. This was on Docker 29.9.2 and 29.1.3. On 27.5.1 the container stays in Created, and I did not test 28.x. So not a Docker bug: the check that can't fail is mine.

The same shape in Ansible. A prometheus-node-exporter from apt held port 9100. The first playbook run failed on the bind. The second one exited 0 with ok=19 changed=0 failed=0, and the node-exporter target was down (Docker 29.9.2). In part 1 I used changed=0 as a check. It proves the run is idempotent, not that anything is running.

smoke-check.sh failed in both states. The docs just never mentioned it.

An upgrade that exits 0

I upgraded from v2.2.0 to v2.3.0 over existing volumes, the obvious way: check out the new tag, docker compose pull, docker compose up -d. Data survived and all 5 targets were up. But Grafana showed 13 dashboards instead of 9: three new ones were not applied, their old versions stayed, plus 4 stale ones. Every 30 seconds the log said failed to save dashboard. up -d exited 0.

It is exactly the three files whose names stayed while the uid inside changed. That fits, but I did not confirm it in Grafana's code. The only fix I verified is deleting the Grafana volume, which also deletes users and anything made in the UI. This one shipped in my own release, v2.3.0.

Two more, in a line each: a new GRAFANA_PASSWORD is ignored after the first start, and Compose and Ansible on one host collide on container names and leave a network, three volumes and /opt/monitoring behind.

What changed

It is in v2.3.1 (PR #45): the docs verify with smoke-check.sh instead of docker compose ps, TROUBLESHOOTING.md has these cases, and a new CI job upgrades the latest release over its own data and checks the dashboards. The release is docs and CI only. I did not touch the healthcheck, the playbook still does not check targets after it starts, and the dashboard bug itself is not fixed: the docs now tell you how to spot it and how to get around it.

Not checked: Docker 28.x, x86_64, Debian and RHEL, the Ansible upgrade path, an alert reaching Telegram. All of the above is one VM.

Do you test installs on a machine that already has leftovers, or only on a fresh one? What do you leave on it?

Repo: github.com/Airat71/monitoring-stack

An AI assistant helped with this article.