AI-assisted: I wrote the outline and the examples, and used an AI tool to tighten the wording. I checked every behaviour below against the Kubernetes documentation. I run LearnSome.tech, where we teach Kubernetes; this post doesn't need anything from it.

Most probe problems I see aren't exotic. A Pod restarts every few minutes, kubectl get pods shows a climbing RESTARTS count, and the app logs look fine. Nine times out of ten the probes are doing exactly what they were told to do. Here are the five mistakes I look for first.

1. Using liveness to protect a slow start

A JVM service takes 70 seconds to start. Someone adds a liveness probe with the defaults (periodSeconds: 10, failureThreshold: 3, no initial delay), and the kubelet kills the container at about 30 seconds, before it ever listens. It restarts, takes 70 seconds again, and is killed again. After a few rounds you get CrashLoopBackOff, with back-off delays doubling from 10 s up to a 5-minute cap.

Fix: add a startup probe. Until it succeeds, Kubernetes doesn't run liveness or readiness at all. Size it so failureThreshold × periodSeconds covers your worst start, not your average one:

startupProbe: httpGet: { path: /healthz, port: http } periodSeconds: 10 failureThreshold: 18 # up to 3 minutes to start

Then liveness can stay strict, because it only begins after the start is done.

2. Checking the database in the liveness probe

If /healthz queries Postgres and Postgres has a bad minute, every replica fails liveness together and Kubernetes restarts all of them. Restarting your app doesn't fix your database; it adds a thundering herd of reconnects on top of the outage.

Fix: liveness answers one question: "is this process stuck?" Keep it in-process and cheap. Put dependency checks in the readiness probe, where a failure only takes the Pod out of the Service's endpoints. Nothing gets killed.

3. Trusting the 1-second timeout

timeoutSeconds defaults to 1. A health endpoint that answers in 40 ms on your laptop can take 1.5 s under load, during a GC pause, or on a noisy node. Three slow answers in a row and the container is restarted, right when it's busiest.

Fix: time the endpoint under realistic load and set timeoutSeconds from that, or make the endpoint cheaper. Health endpoints should return a tiny body; the kubelet stops reading after 10 KiB anyway.

4. Expecting liveness to wait for readiness

It doesn't. Liveness and readiness run independently. A Pod can be "not ready" and still have its liveness probe firing, and failing, the whole time. People add a readiness probe and assume it "protects" the container during warm-up. It doesn't protect it from liveness.

Fix: that's the startup probe's job again (mistake 1), or an initialDelaySeconds on liveness if you really can't add one.

5. Copying the same probe for all three

Same path, same port, same thresholds, pasted three times. It looks tidy, but it erases the difference between "restart me", "don't send me traffic" and "I'm still booting".

Fix: a reasonable starting point:

  • Startup: the same cheap check as liveness, with a long failureThreshold.
  • Liveness: cheap, in-process, a few failures before a restart.
  • Readiness: its own endpoint, may check required dependencies, fails fast (periodSeconds: 5, failureThreshold: 2).

How I debug it 

kubectl describe pod <pod> # Events: "Liveness probe failed: ..." and "Killing" kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState}' kubectl get endpointslices -l kubernetes.io/service-name=<svc> # is the Pod in the Service?

If the Events show "Liveness probe failed" just before each restart, it's the probe and not the app. If the Pod is running but gets no traffic, look at readiness and the EndpointSlices.

One habit worth building: before you ship a new liveness probe, run it against a deliberately slow copy of the app. If that copy gets restarted, so will production on a bad day.

What's the strangest probe failure you've had to debug?