A backup script on one of my boxes ran every night for three weeks. It exited 0. It wrote nothing. Nothing alerted, and from the system's perspective, nothing had failed.

I found out because I needed a file from six months ago.

This is not an unusual story. If you have run servers long enough, you have your own version of it. What made it interesting to me was how many different ways the same class of failure can hide, and how little of it is covered by the monitoring people typically install.

The failure that doesn't look like a failure

The mental model most of us carry is: backup → error → alert. But the failures that actually bite are the ones where the exit code is zero.

  • The target path moved because a volume got remounted somewhere else, so rsync cheerfully copied nothing.
  • The destination filled up, and the tool wrote a zero-byte file and called it a day.
  • A cron entry was dropped during an OS upgrade. The job did not fail — it stopped existing.
  • The script succeeded, but the snapshot retention logic deleted the new snapshot.
  • systemd put the unit into a failed state weeks ago and nobody looked.

In none of these does the thing you are monitoring — "did the process exit non-zero" — fire.

So the question has to change shape. Instead of "did the job succeed", ask "how old is the newest artifact". That question is answerable without trusting the job at all:

find /backups -maxdepth 1 -type f -printf '%T@ %p\n' | sort -n | tail -1

If the newest file is older than your RPO, you are in trouble regardless of what any exit code said. That one inversion is most of the value.

The second thing I got wrong: df -h

df -h said 40% free. The application still could not write, because inodes were exhausted — a mail queue had accumulated a few million tiny files.

df -h and df -i answer different questions, and the second one fails harder when it fails:

df -P -x tmpfs -x devtmpfs | awk 'NR>1 && $5+0 > 85' df -Pi -x tmpfs -x devtmpfs | awk 'NR>1 && $5+0 > 85'

A related trap: percent-used is the wrong metric for alerting. A volume growing at 2 GB/hour is an emergency at 60%; one sitting at 84% for two years is not. What you want is time to full:

time_to_full ≈ (size_bytes − used_bytes) / growth_bytes_per_hour

That requires keeping a little history, which is exactly what people skip because the dashboard already "shows disk".

Clock skew, the silent one

This one surprised me. A box's clock drifted a few minutes. Nothing crashed. But:

  • TLS validation against external endpoints began failing intermittently
  • cron fired at surprising times, because it works off wall-clock time
  • Correlating logs across machines became fiction

timedatectl will tell you whether the clock is synchronised. One line, almost nobody has the check, and when it is wrong it makes every other investigation harder.

systemctl --failed is criminally underused 

$ systemctl --failed  UNIT LOAD ACTIVE SUB DESCRIPTION ● backup@daily.service loaded failed failed Daily backup

A unit can sit in failed for months. It is not in your logs unless you are looking, and if it is the backup unit, the backup has been dead the whole time while the timer rescheduled itself every night.

What I ended up building

I wanted one command that answered: is anything about to break, and did last night's backup actually run? Ten read-only checks, aggregated:

$ vaultcheck audit  Vaultcheck audit — web-01 — overall: CRIT CRIT backup newest artifact is 22d 4h old (threshold 26h) WARN disk /var at 87% (5.2 GB free) OK inodes max 31% (/) OK memory 1.4 GB / 8.0 GB used OK load 0.42 / 0.31 / 0.28 CRIT systemd backup@daily.service failed OK cron crontab present, 6 jobs WARN clock not synchronised (drift ~184s) OK logs log growth within limits WARN updates 14 pending, reboot required

Design constraints, and why

Standard library only. No pip, no virtualenv, no C extension build step. The artifact is a single 38 KB .pyz you can scp. Any dependency becomes a liability when you are installing something on thirty boxes, and a monitoring tool that can itself fail to install is a bad joke.

Read-only, no agent, no daemon. Nothing runs between invocations. There is no state to corrupt, nothing to keep alive, nothing to fight with your init system. You can verify that claim cheaply: run --only disk and watch what happens.

Exit codes designed for automation. --fail-on warn|crit makes it exit non-zero at a chosen severity, so it composes with cron, systemd timers or a heartbeat service instead of asking you to look at something.

No telemetry. No phone-home, no account, no licence check in the audit path. A tool that inspects your servers should be inspectable itself.

Output formats 

vaultcheck audit --format text # humans vaultcheck audit --format json # feed into anything vaultcheck audit --format html -o report.html

JSON is the one I actually use: it lets me diff yesterday against today and only alert on change, which kills the "alert fires every night forever" failure mode that trains people to stop reading alerts.

What it isn't

Worth being blunt about, because I have seen these tools oversold:

  • Not a replacement for Prometheus, Zabbix or Netdata. No time series, no graphing, no long-term storage.
  • Linux only.
  • Closed source, and paid — one-off price, no subscription.

The tradeoff I am making is "one command, no dependencies, works in ten seconds on a box you just SSH'd into" versus "fits into a mature observability stack". If you already run Prometheus with node_exporter and alert rules for backup freshness, you do not need this, and I would genuinely rather you use what you have.

If you do not, the free version of 80% of it is:

# backup freshness (oldest acceptable: 26h) find /backups -maxdepth 1 -type f -printf '%T@ %p\n' | sort -n | tail -1 # disk and inodes df -P -x tmpfs -x devtmpfs | awk 'NR>1 && $5+0>85' df -Pi -x tmpfs -x devtmpfs | awk 'NR>1 && $5+0>85' # failed units, cron presence, clock systemctl --failed systemctl is-enabled cron 2>/dev/null || systemctl is-enabled cronie 2>/dev/null timedatectl | grep -i synchronized

The one thing I would tell my past self

Assert on the absence of good output, not the presence of errors.

Every failure in this article — the empty backup, the full inodes, the dead cron entry, the failed unit — is invisible if your only signal is "did the process complain". Make the happy path the thing you verify, and monitor for its absence.