Our monitor said 100% uptime while the CDN served a 6-hour-old homepage

This is a hypothetical postmortem, stitched together from failure patterns that show up again and again. The names are made up. The mechanics are not.

At 09:14 on a Tuesday, a customer emailed support: "Your pricing page says the sale ends today, but it ended yesterday." Support forwarded it to engineering. Engineering checked the dashboard. Uptime: 100%. Response time: 38 ms, faster than usual.

That speed should have been the first warning sign.

What actually happened

• 03:02: a database migration locked a table. The app servers started timing out on every request that touched it, which was every page render.
• 03:02 onward: the CDN in front of the site was configured with stale-if-error and a generous stale-while-revalidate. When the origin returned 504s, the CDN served the last good copy. That was the right call from the CDN's point of view.
• 03:02–09:14: the uptime monitor requested https://example.com/ every minute. It got a 200 OK from the edge each time, in 38 ms, because nothing was being generated. Green across the board.
• 09:14: a human noticed the content was old.

Six hours of "up", and no new signups, because the signup form posted to the origin and failed. The homepage looked fine.

Why every check passed

The monitor asked one question: does this URL return 2xx? The CDN answered it honestly. The real question was: is the application working? Those are different questions, and the CDN sat between them.

Three things made it invisible:

1. Checking a cacheable URL. The homepage is the most cached page on most sites. It's the worst place to look for origin health.
2. No content assertion. The check didn't look at the body at all.
3. No freshness signal. Nothing in the response said when the page had been generated.

The fixes (with code)

1. Check a URL the CDN won't cache

Add a dedicated endpoint that always hits the origin and does real work:

# Flask example from flask import Flask, jsonify import time app = Flask(__name__) @app.get("/healthz/deep") def deep_health(): t0 = time.time() db.session.execute("SELECT 1 FROM users LIMIT 1") # touch a real table  resp = jsonify(ok=True, db_ms=round((time.time() - t0) * 1000)) resp.headers["Cache-Control"] = "no-store, private" return resp 

no-store matters. Some CDNs cache 200s by default unless you tell them not to. While you're there, double-check that your CDN rules don't override origin headers for /healthz*.

2. Expose a generation timestamp and assert on it

Have the origin stamp every HTML response:

# nginx in front of the app add_header X-Generated-At $msec always; 

Or do it inside the template: <meta name="generated-at" content="{{ now_epoch }}">. Then check freshness from outside:

#!/usr/bin/env bash # freshness.sh: fail if the page is older than MAX_AGE seconds URL="https://example.com/" MAX_AGE=900 gen=$(curl -s -D - -o /dev/null "$URL" | awk -F': ' 'tolower($1)=="x-generated-at"{print int($2)}') now=$(date +%s) if [ -z "$gen" ]; then echo "no X-Generated-At header"; exit 2; fi age=$(( now - gen )) echo "page age: ${age}s" [ "$age" -le "$MAX_AGE" ] || { echo "STALE"; exit 1; } 

On a busy site with a short TTL, the page's age should stay under a few minutes. If it climbs to hours, the origin isn't producing pages, whatever the status code says.

3. Read the CDN's own cache headers

Most CDNs tell you what they did. Cloudflare sends cf-cache-status, Fastly and Varnish send x-cache and age, CloudFront sends x-cache: Hit from cloudfront. The Age header is the useful one:

curl -sI https://example.com/ | grep -iE '^(age|cf-cache-status|x-cache):' 

An Age of 21600 on a page with a 60-second TTL means you're looking at stale-if-error in action.

4. Monitor the write path, not just the read path

The real damage was failed signups. A synthetic check that submits a form to a test endpoint (or calls the same API the form calls) would have gone red at 03:03. If a full transaction is too heavy, at least POST to an endpoint the CDN can't cache. Most CDNs pass non-GET methods straight through to the origin.

What we'd tell another team

Stale-if-error is good. Keep it. It turned a hard outage into a soft one for readers. The mistake was letting it hide the outage from us as well.

A fast response is a symptom too. If p50 latency suddenly halves and nobody shipped a performance fix, something is probably being served from somewhere it shouldn't be. Alert on sudden drops, not just spikes.

Check from outside, at two layers. One check for "can users load the page" (the cached homepage is fine for that) and one for "is the origin alive" (the uncached deep health endpoint). Failures where the two disagree are exactly the ones you want to know about.

A simple outside-in check, like the ones you can run on upordown.online, tells you whether the site is reachable for everyone. Pair it with an origin-level check and you cover both sides of the CDN.

A minimal monitoring matrix

Four checks, maybe twenty minutes of setup. For six hours of silent failure, that's a cheap trade.

Related reading

• Your health check returns 200 while users get errors: 7 health check mistakes
• 503 vs 504: what each error means and who has to fix it
• Ping is not an uptime check: 5 ways ICMP lies about whether your site is up
• A free uptime check with GitHub Actions that opens (and closes) an issue for you

Have you ever had a CDN, a cache or a service worker cover up an outage so well that you only found out from a customer? How did you catch it next time?