The backup check that says fresh because it could not read the directory

A backup checker I wrote printed this, on a backup target that no longer existed in any readable sense:

ok | last newest entry 0.0 h old 

chmod 000 on that directory. os.listdir raised PermissionError for the user the check runs as. And the report said the newest backup was written just now, with no hedge of any kind, along with eleven other green lines and the summary All backup targets are fresh.

The directory was, at that moment, the only thing the check had actually looked at.

The eight lines responsible

newest = None for root, dirs, files in os.walk(path, onerror=lambda _e: None): for name in files: try: m = os.stat(os.path.join(root, name)).st_mtime except OSError: continue # every file unreadable -> skip if newest is None or m > newest: newest = m if newest is None: newest = os.stat(path).st_mtime # <- the fallback that lies return (time.time() - newest) / 3600.0 

Two lines carry the bug. The onerror handler is the first: os.walk ignores scandir errors by default, so onerror=lambda _e: None is a no-op wearing the costume of error handling. It looks like somebody thought about failure. Nothing is signaled either way — a walk over an unreadable directory yields nothing, exactly like a walk over an empty one.

The fallback is the second: when the walk found no files, the code asks the directory for its own mtime and treats that as the age of the newest artifact.

That fallback is not a reasonable guess. It answers a different question, and it answers it with a number that is always plausible.

What the permission bits actually gate

I reproduced it on a plain Linux box today, dropping to nobody before measuring, because as root every case below just works and proves nothing:

stat(execonly dir) [0111] ok stat(execonly/db.dump) [0111] ok listdir(execonly) [0111] PermissionError stat(noperm dir) [0000] ok stat(noperm/db.dump) [0000] PermissionError listdir(noperm) [0000] PermissionError os.walk(noperm) files: default=0 onerror=lambda:None=0 

Read it in order, because the four lines tell the whole story:

• os.stat on a directory needs execute permission on its parent and nothing else. Mode 0000 on the directory itself does not stop it.
• os.stat on a file inside a directory needs execute on that directory — you can stat a known name through an execute-only parent, and 0111 is enough.
• os.listdir needs the read bit. That is the one operation that fails.
• os.walk swallows it either way, so "unreadable" and "empty" arrive at the caller as the same value: zero files.

So the fallback was guaranteed to succeed precisely in the case it was written for — the case where the walk had nothing to report. The failure mode is not "the code crashed on an edge case". The code takes the branch that cannot fail, and produces a fresh timestamp from a directory it could not open.

A directory's mtime is not a backup timestamp

Even with a readable directory, the fallback is answering the wrong question. Measured on the same box:

after create file dir mtime moved after rewrite content dir mtime unchanged, file mtime moved after rename dir mtime moved 

A directory's mtime tracks changes to its entry list — create, unlink, rename. It does not move when a file's contents are rewritten in place. Which is the normal shape of nightly backups: pg_dump --file /backup/db.dump overwrites the same name every night, rsync into a fixed target, tar czf onto the same path. In that workload the directory mtime is the date the target was created, not the date of the last successful write.

Three of the four cases I measured were wrong, and they were wrong in opposite directions — all from the same branch:

The third row is the one that gets missed. A false red costs an argument with the on-call engineer, who will usually be right, and after the second argument the check gets muted — which is why a checker that lies in both directions ends up trusted in neither.

Three answers, not one

The fix is not a better fallback. It is refusing to compress three states into one:

cannot read directory -> CRIT "cannot read <path>: [Errno 13] Permission denied" readable but empty -> CRIT "no files found under <path>" readable with files -> only now, the freshness comparison 

"I cannot see it" is never allowed to be translated into a statement about freshness in either direction. Saying "I don't know" out loud is what makes the rest of the report worth reading; borrowing "stale" or "fresh" to fill the gap means the reader trusts a claim no code ever made.

The test that holds it in place is three directories and one assertion per state, run as a non-root user:

mkdir -p /tmp/vc/{fresh,empty,locked} touch /tmp/vc/fresh/db.dump chmod 000 /tmp/vc/locked su nobody -s /bin/sh -c 'your-check /tmp/vc/fresh /tmp/vc/empty /tmp/vc/locked' 

• /tmp/vc/locked must exit non-zero with an error naming the errno, not a freshness verdict.
• /tmp/vc/empty must exit non-zero saying nothing was found.
• /tmp/vc/fresh must be ok, and only that one.

If your checker runs as root in a container, run that last command as nobody or your test proves nothing at all — root ignores every permission bit you set. That mistake costs more audits than it should, including one of mine.

What this is not

It is not a story about a typo. Every line was written by someone trying to avoid a crash, and the result was a checker whose green light meant "I have no idea" in three different situations. A monitoring system earns its keep on the day something breaks, and the only thing that breaks it irreparably is a green tick that was never earned — because the tick is what stops the human from walking over to look.

That specific check is what I sell now, and the reason the tool exists is that this exact path exists in a lot of home-grown backup scripts:

https://vaultcheck.dafili.cn/