The email that ended it was four words long: "is the digest dead?"

Fair question. The page had been loading fine every single morning. Header, date, footer, all rendered. Between the header and the footer: "No new items today." Forty-one days in a row, on a page that had been averaging 31 items a day since March.

Nothing had crashed. No 500s, no restarts, no alerts. My uptime monitor was green the entire time — correctly, because the page really was up. It just had nothing to say.

The job was fine too

That's the part that took me longest to accept. agent_digest.py ran every morning at 6:00 and exited 0. I could see it in the log:

2026-09-14 06:00:02 INFO collected 34 items 2026-09-14 06:00:03 INFO wrote /srv/agent/out/2026-09-14.json 2026-09-14 06:00:03 INFO done

Thirty-four items. Written. Done. Every day. The files existed. I could cat them and see the JSON.

So for the first two weeks I was certain the bug was in the reader. I rewrote the glob. I added caching, then removed the caching. I checked the timezone handling on the date comparison. All of it was fine, because the reader wasn't broken — it was being told "no" by the operating system, and my code was politely throwing that answer away.

Where the exception went

This is the reader, minus the comments I'd added to make it look responsible:

def load_digests(days=7): try: files = sorted(OUT.glob("*.json"))[-days:] return [json.loads(f.read_text()) for f in files] except Exception: return []

except Exception: return []. I wrote that eleven months ago, on a Friday, because a half-written file from a crashed run had 500'd the page in front of a customer. It fixed the symptom in four minutes and then sat there quietly converting PermissionError into "the world is empty" for the next 41 days.

The exception was logged nowhere. Not to the app log, not to stderr, not to a counter. A bare except with a fallback value is not error handling — it's a silent data-loss funnel, and mine had a 41-day backlog.

Why my own verification lied to me

Here's the detail I'm least proud of. During that month I had a root shell open in a tmux window on the Pi — I was doing package upgrades and a systemd migration, and it was just easier to leave it there.

Every check I ran, I ran as root:

ls -l /srv/agent/out | tail -3 -rw------- 1 root root 18K Nov 12 06:00 2026-11-12.json

I looked at that line at least a dozen times. I read it as "the file is there, so the reader can read it." The permission bits were in the output the whole time, on the same line, and I never once read them. I'd internalized -rw------- as normal because every file I created all month looked like that.

My health check had the same blind spot. It ran from the same crontab, as the same user:

open(newest_file).read() # passes, because it's root

So the writer verified itself, as itself, in the same privileged context, and reported success. 1,271 items went into 41 files that the web app had never been able to open.

The two commands that found it in ninety seconds

Once the customer email made me actually look, it took less time than making coffee.

# 1. Check ownership drift in the output dir sudo find /srv/agent/out -maxdepth 1 -user root -newermt '-60 days' | wc -l 41 # 2. Try to read as the user the app actually runs as sudo -u app python3 -c " import pathlib p = sorted(pathlib.Path('/srv/agent/out').glob('*.json'))[-1] print(p.read_text()[:60]) " PermissionError: [Errno 13] Permission denied

The first command found 41 files owned by root. The second reproduced the exact exception the bare except had been eating since September.

The root cause was two decisions, made on different days, that combined badly:

  1. During a hardening pass I added os.umask(0o077) to the agent — correct, files should be private.
  2. Six weeks earlier, at 1 AM, debugging a missing venv, I'd installed the job with sudo crontab -e instead of the app user's crontab, and never moved it.

Either one alone was harmless. Together they meant every output file was 0600 root:root while the reader ran as app.

The fix, in three layers

Layer 1 — the job no longer runs as root. It doesn't need to. It needs write access to one directory.

# /etc/systemd/system/agent-digest.service [Service] User=agent Group=agent UMask=0027 ExecStart=/srv/agent/venv/bin/python /srv/agent/agent_digest.py

Output is now 0640 agent:agent, the directory is 2750 with group agent, and app is a member of that group. Nothing needs sudo. The unit also gives me Restart=on-failure and a real exit code I can alert on, which crontab never did.

Layer 2 — the smoke test runs as the reader, in the deploy script.

sudo -u app python3 -c " import pathlib p = sorted(pathlib.Path('/srv/agent/out').glob('*.json'))[-1] print(p.read_text()[:40] and 'READABLE') " || exit 1

This is the check that would have caught it on day one, and it's one line in the deploy. Nothing else in my setup tests the boundary that actually failed.

Layer 3 — the exception is logged, and volume is monitored.

def load_digests(days=7): files = sorted(OUT.glob("*.json"))[-days:] out = [] for f in files: try: out.append(json.loads(f.read_text())) except (OSError, json.JSONDecodeError) as e: log.warning("digest read failed: %s: %s", f.name, e) return out

Now a bad file costs me one item, not the whole page, and the failure leaves a fingerprint. I also alert on content volume, not just liveness: if len(items) == 0 two days running, or the newest file is older than 26 hours, I get a push. Uptime monitoring can't see an empty page — it can only see whether something answered.

What I actually learned

Verify as the consumer, never as the producer. The writer, the verifier, and the reader all ran as root, so all three agreed and all three were wrong. Any check that runs in the same privilege context as the thing it's checking is a check of your intentions, not of your system.

A fallback value hides a whole category of bug. return [] on exception feels defensive. In practice it converts every environment problem — permissions, missing mounts, full disks, corrupt files — into the same indistinguishable "there's no data today," and that state looks exactly like a genuinely quiet day.

Least privilege doesn't fail loudly, it fails forty days later. The permission change was correct and the cron entry was convenient, and nothing complained for six weeks. The failure surfaced as a customer noticing a product decision I never made.

ls -l puts the answer on the same line as the reassurance. Ownership and mode were visible in every check I ran. I read the filename and the size, because that's what I was looking for. Once a month, print the bits you're not looking for.

If your agent writes files another process reads

  • sudo find <out_dir> -user root -newermt '-30 days' — ownership drift is a one-liner, not a hunt
  • sudo -u <reader_user> test -r <file> — test the boundary, not the file
  • grep your codebase for except: and except Exception: with a fallback return; every one is a future silent outage
  • alert on volume and freshness, not just HTTP status
  • make the writer an unprivileged, dedicated user, and set UMask= in the unit instead of trusting the shell
  • if a job runs from a crontab, ask whose crontab — check sudo crontab -l as well as crontab -l

Forty-one days, 1,271 items, zero errors in any log I was watching, and a customer had to tell me.

The full checklist + scripts are in Ship Safe — The Launch-Day Security Kit — code LAUNCH90 at checkout makes it $1.50.