Kubernetes advertises rolling updates out of the box, yet many teams still see a burst of 502s on every deploy. The defaults are only part of the story: zero downtime needs correct probes, a sensible rollout strategy, graceful shutdown, and protection from voluntary disruptions like node drains. This post walks through each piece with manifests you can apply today.

Why requests get dropped

During a rollout, three things can go wrong:

  1. New pods receive traffic before they are ready. The container process is running, but caches, DB pools, or JIT warm-up are not done.
  2. Old pods are killed while they still have in-flight requests. Endpoint removal and SIGTERM happen concurrently, so a pod may receive new connections after it starts shutting down.
  3. Too many pods disappear at once. A rollout combined with a node drain or autoscaler scale-down can drop capacity below what your traffic needs.

Each problem has a dedicated Kubernetes feature.

Step 1: Readiness, liveness, and startup probes

  • Readiness decides whether a pod is in the Service's endpoints. Fail it and traffic stops; the pod is not restarted.
  • Liveness decides whether the container should be restarted. Use it only for "this process is wedged" conditions.
  • Startup holds off liveness checks until a slow-starting app has booted. 
apiVersion: apps/v1 kind: Deployment metadata: name: web labels: app: web spec: replicas: 4 revisionHistoryLimit: 5 selector: matchLabels: app: web strategy: type: RollingUpdate rollingUpdate: maxSurge: 25% maxUnavailable: 0 minReadySeconds: 10 template: metadata: labels: app: web spec: terminationGracePeriodSeconds: 45 containers: - name: web image: ghcr.io/example/web:1.4.2 ports: - name: http containerPort: 8080 resources: requests: cpu: 250m memory: 256Mi limits: memory: 512Mi startupProbe: httpGet: path: /healthz port: http periodSeconds: 5 failureThreshold: 24 # up to 120s to boot readinessProbe: httpGet: path: /ready port: http periodSeconds: 5 timeoutSeconds: 2 failureThreshold: 2 livenessProbe: httpGet: path: /healthz port: http periodSeconds: 10 timeoutSeconds: 2 failureThreshold: 3 lifecycle: preStop: exec: command: ["sh", "-c", "sleep 10"]

Keep /healthz cheap and local: it should answer "is my process alive?", not "is the database up?". If liveness checks a shared dependency, a database blip restarts every pod at once. /ready can check dependencies the pod truly can't serve without.

Step 2: A rollout strategy that never reduces capacity

The two knobs that matter:

  • maxUnavailable: 0 means Kubernetes never removes an old pod until a new one is Ready.
  • maxSurge: 25% lets it create extra pods (here, 1 of 4) above the desired count during the rollout.

minReadySeconds: 10 requires a new pod to stay Ready for 10 seconds before it counts as available. That catches apps that pass their first readiness check and then crash.

The trade-off is that you need spare cluster capacity for the surge pods. If your cluster is tightly packed, the rollout stalls in Pending instead of dropping traffic, which is the failure mode you want.

Step 3: Graceful shutdown

When a pod is deleted, Kubernetes sends SIGTERM and, in parallel, removes the pod from Service endpoints. kube-proxy, ingress controllers, and cloud load balancers take a few seconds to notice. The preStop sleep above keeps the pod serving during that window; only then does SIGTERM reach your process.

Your application must also handle SIGTERM by finishing in-flight requests. A minimal Python example:

import signal import sys from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler import threading class Handler(BaseHTTPRequestHandler): def do_GET(self): body = b"ok" self.send_response(200) self.send_header("Content-Length", str(len(body))) self.end_headers() self.wfile.write(body) server = ThreadingHTTPServer(("0.0.0.0", 8080), Handler) def shutdown(signum, frame): # shutdown() blocks until serve_forever exits, so run it in a thread  threading.Thread(target=server.shutdown, daemon=True).start() signal.signal(signal.SIGTERM, shutdown) server.serve_forever() server.server_close() sys.exit(0)

Make sure terminationGracePeriodSeconds exceeds the preStop sleep plus your longest request. Otherwise the kubelet sends SIGKILL mid-request.

Step 4: PodDisruptionBudgets for voluntary disruptions

Rollouts aren't the only time pods go away. kubectl drain, cluster upgrades, and the cluster autoscaler all evict pods. A PodDisruptionBudget (PDB) tells the eviction API how many pods must stay up:

apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: web-pdb spec: minAvailable: 3 selector: matchLabels: app: web

With 4 replicas and minAvailable: 3, a drain evicts at most one web pod at a time and waits for its replacement to become Ready. Spread the replicas across nodes so a single drain doesn't touch most of them:

 topologySpreadConstraints: - maxSkew: 1 topologyKey: kubernetes.io/hostname whenUnsatisfiable: ScheduleAnyway labelSelector: matchLabels: app: web

(Add that under spec.template.spec in the Deployment.)

Step 5: Deploy and verify with kubectl 

#!/usr/bin/env bash set -euo pipefail NS="${NS:-default}" IMAGE="${1:?usage: deploy.sh <image>}" kubectl -n "$NS" set image deployment/web web="$IMAGE" kubectl -n "$NS" annotate deployment/web \ kubernetes.io/change-cause="deploy $IMAGE" --overwrite if ! kubectl -n "$NS" rollout status deployment/web --timeout=300s; then echo "Rollout failed, rolling back" >&2 kubectl -n "$NS" rollout undo deployment/web kubectl -n "$NS" rollout status deployment/web --timeout=300s exit 1 fi kubectl -n "$NS" get pdb web-pdb

Set spec.progressDeadlineSeconds (default 600) so a stuck rollout is reported as failed and rollout status exits non-zero.

To prove zero downtime, run a load loop against the Service while deploying:

while true; do code=$(curl -s -o /dev/null -w '%{http_code}' http://web.default.svc.cluster.local/) [ "$code" != "200" ] && echo "$(date +%T) got $code" sleep 0.1 done

Any line printed during a rollout is a dropped request worth investigating.

Common pitfalls

  • Liveness probe identical to readiness, checking the database. One dependency outage becomes a cluster-wide restart storm.
  • No preStop delay. Traffic still routes to terminating pods for a few seconds, causing connection resets.
  • minAvailable equal to replicas. The PDB blocks every drain forever, and node upgrades hang.
  • Single replica. No probe or PDB can make one pod zero-downtime. Run at least two.
  • Shells that swallow signals. command: ["sh", "-c", "python app.py"] may leave sh as PID 1, and it won't forward SIGTERM. Use exec or an init like tini.
  • Probe timeouts too tight. A 1s timeout on a GC-heavy JVM causes flapping under load.

Wrap-up

Zero-downtime deploys come from layers that work together: probes gate traffic, maxUnavailable: 0 preserves capacity, preStop plus SIGTERM handling drains connections, and PDBs guard against drains. Add the curl loop to your staging pipeline and you'll catch regressions before users do.

Written by the ByteBridge Talent LLC engineering team.