Your health check returns 200 while users get errors: 7 health check mistakes

The dashboard was green. The load balancer was happy. Every /health probe returned 200 OK in 4 ms. Meanwhile, every checkout request was failing because the database connection pool was exhausted, and nobody found out until support tickets started piling up.

If that sounds familiar, the health check wasn't broken. It was answering a different question than the one you cared about.

TL;DR

• A health check that only proves the process is alive will stay green through most real outages.
• Split it in two: a cheap liveness check and a readiness check that touches real dependencies with tight timeouts.
• Monitor from the outside on a real user path too. Your own /health can't see DNS, TLS or CDN problems.
• My opinion: a single /health endpoint used for everything is a design smell.

Mistake 1: returning a hardcoded 200

app.get('/health', (req, res) => res.send('ok')); 

This proves Node is running and the router works. It says nothing about the database, the cache, the disk or the queue. It is a fine liveness check and a useless readiness check.

Mistake 2: using one endpoint for both liveness and readiness

Orchestrators treat these differently. A failed liveness probe restarts the container. A failed readiness probe stops sending traffic to it.

If your one endpoint checks the database and the database blips, Kubernetes restarts every pod at once. Now you have a database blip and a cold start for the whole fleet.

livenessProbe: httpGet: { path: /livez, port: 8080 } periodSeconds: 10 failureThreshold: 3 readinessProbe: httpGet: { path: /readyz, port: 8080 } periodSeconds: 5 failureThreshold: 2 

Keep /livez dumb. Put dependency checks only in /readyz.

Mistake 3: dependency checks with no timeout

A readiness check that calls the database with the driver's default timeout can hang for 30 seconds. The probe times out, but your handler keeps a connection open, and under load the health checks themselves eat the pool.

app.get('/readyz', async (req, res) => { const checks = { db: withTimeout(db.query('SELECT 1'), 500), cache: withTimeout(redis.ping(), 300), }; const results = {}; let ok = true; for (const [name, p] of Object.entries(checks)) { try { await p; results[name] = 'ok'; } catch (e) { results[name] = e.message; ok = false; } } res.status(ok ? 200 : 503).json(results); }); function withTimeout(promise, ms) { return Promise.race([ promise, new Promise((_, rej) => setTimeout(() => rej(new Error(`timeout ${ms}ms`)), ms)), ]); } 

Short timeouts, a 503 when something is wrong, and a JSON body that tells the on-call person which dependency failed.

Mistake 4: failing readiness on optional dependencies

If the recommendations service is down, can users still buy things? Then it should not fail readiness. Pulling every pod out of rotation because a non-critical sidecar is slow turns a degraded feature into a full outage.

Sort dependencies into two lists: must have (fail readiness) and nice to have (report as degraded, still return 200). Be honest about which list each one belongs in.

Mistake 5: caching the health response for too long

Some teams cache the health result to avoid hammering the database. Reasonable, but a 60-second cache plus a 3-failure threshold at 10-second intervals means you can serve errors for 90 seconds before anything reacts. A cache of 2 to 5 seconds is usually enough to protect the database.

Mistake 6: only checking from inside the network

Your internal probe goes pod to pod. It never touches:

• public DNS
• the TLS certificate users see
• the CDN or WAF in front of you
• the route from other countries

All of these break regularly. An expired certificate gives a perfectly green internal check. If you've never debugged a request layer by layer, the curl trace walkthrough in How to read curl -v output to find where a request fails shows which layer each failure lives in.

A quick outside-in check you can run from any machine:

curl -sS -o /dev/null --max-time 10 \ -w 'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} total=%{time_total} code=%{http_code}\n' \ https://example.com/checkout 

Run it on a real page, not /health. That gets you closer to what a user sees.

Mistake 7: treating "the check passed" as "the site is up"

A 200 with an empty page, a maintenance banner or a "something went wrong" template is still a 200. Check for content too:

body=$(curl -sS --max-time 10 https://example.com/) echo "$body" | grep -q 'Add to cart' && echo UP || echo "DOWN (content missing)" 

Keyword checks catch the "200 but broken" failures that status codes miss. They also catch the case where a CDN serves a stale error page with a cheerful status code.

A health check that tells the truth

Putting it together:

The last row is the one people skip, and it's the one that matches what users actually experience. When users say "it's down" and your dashboards say "it's fine", a neutral outside check such as upordown.online is a fast way to settle whether the site is down for everyone or just for some people.

My take

Teams should stop arguing about "deep vs shallow" health checks as if one were correct. You need both, and they answer different questions. The shallow one decides "should this container be restarted?". The deep one decides "should this instance get traffic?". An outside check on a real page decides "should a human wake up?". Mixing those three questions into one endpoint is where most green-dashboard outages come from.

If you want a follow-up on what to do once a check fails, the next post covers why ping isn't an uptime check, and for the communication side, see How to write status page updates people can actually use.

Over to you

Does your readiness check include the database, or do you keep it out on purpose to avoid fleet-wide flapping? I'd like to hear where you drew the line and what outage taught you to draw it there.