Zero-Downtime Graceful Shutdown in Kubernetes: SIGTERM, In-Flight Draining

Introduction

Imagine running a production application on Kubernetes that handles thousands of requests every minute.

Your application is healthy, users are actively interacting with it, and everything appears to be working perfectly. Suddenly, Kubernetes needs to restart one of your application pods because of a deployment, scaling operation, node maintenance, or configuration update.

The pod receives a termination signal and begins shutting down.

But what happens to the requests that are currently being processed?

What happens if a user is uploading a file, an API request is taking several seconds, or a database transaction is still running?

If the application terminates immediately, users may experience:

• HTTP 502 or 503 errors.
• Connection resets.
• Failed API requests.
• Interrupted file uploads.
• Incomplete database transactions.
• Lost messages during Kafka processing.
• Unexpected application downtime during deployments.

This is where Graceful Shutdown in Kubernetes becomes essential.

Graceful shutdown ensures that an application stops accepting new requests, completes existing work, releases resources, and terminates safely.

In this article, we will understand how Kubernetes handles pod termination, how SIGTERM works, how in-flight request draining happens, and how to implement a production-ready zero-downtime shutdown strategy.

If you want to explore more practical backend engineering and DevOps concepts, visit DevCampus, where we publish technical guides covering Kubernetes, Docker, distributed systems, databases, and system design.

1. What Is Graceful Shutdown?

Graceful shutdown is the process of stopping an application without abruptly interrupting its ongoing operations.

Instead of immediately killing a running process, the application follows a controlled shutdown sequence.

A typical graceful shutdown looks like this:

1. Kubernetes decides to terminate a pod.
2. The application receives a SIGTERM signal.
3. The application stops accepting new requests.
4. The application becomes unavailable for new traffic.
5. Existing requests are allowed to complete.
6. Background workers finish or safely stop their current tasks.
7. Database connections and other resources are closed.
8. The application exits successfully.

The objective is to avoid disrupting users while the application is being replaced or stopped.

Graceful Shutdown vs. Immediate Shutdown

Graceful shutdown is particularly important for applications running behind Kubernetes Services, Ingress controllers, API gateways, and load balancers.

2. Understanding Kubernetes Pod Termination Lifecycle

To implement graceful shutdown correctly, we first need to understand what Kubernetes does when a pod is terminated.

Consider a deployment containing three application replicas:

 Load Balancer | Kubernetes Service | +------------+------------+ | | | Pod-1 Pod-2 Pod-3 Running Running Running 

Now imagine deploying a new application version.

Kubernetes needs to terminate an old pod and replace it with a new one.

The termination process involves several important stages.

Step 1: Pod Deletion Is Requested

A deployment controller or another Kubernetes operation requests pod termination.

For example:

kubectl delete pod application-pod 

Kubernetes starts the pod termination process.

The pod receives a deletion timestamp and enters the termination lifecycle.

Step 2: The Termination Grace Period Starts

Kubernetes uses the terminationGracePeriodSeconds setting to determine how much time the container has to shut down.

Example:

spec: terminationGracePeriodSeconds: 60 

This gives the pod up to 60 seconds to terminate gracefully.

If the process does not exit within this period, Kubernetes can forcefully terminate it using SIGKILL.

Step 3: The PreStop Hook Executes

If a preStop lifecycle hook is configured, Kubernetes executes it before sending SIGTERM to the container process.

Example:

lifecycle: preStop: exec: command: - /bin/sh - -c - sleep 10 

This example introduces a 10-second delay.

It can provide additional time for endpoint removal and traffic propagation, depending on the networking and load-balancing architecture.

However, a sleep command alone does not guarantee graceful shutdown.

The application must still handle SIGTERM correctly.

Step 4: Kubernetes Sends SIGTERM

After the preStop hook completes, Kubernetes sends SIGTERM to the container's main process.

SIGTERM is a termination request.

It tells the application:

"Start shutting down safely."

The application should respond by:

• Stopping new requests.
• Draining existing connections.
• Completing active operations.
• Stopping background workers.
• Releasing resources.

Step 5: SIGKILL If the Grace Period Expires

If the process is still running after the termination grace period expires, Kubernetes forcefully terminates it.

SIGKILL cannot be caught or handled by the application.

This means that graceful shutdown must finish within the available termination window.

Important: The termination grace period includes the time spent executing the preStop hook. It is not an additional period granted after SIGTERM.

3. What Is SIGTERM and Why Does Kubernetes Use It?

SIGTERM stands for Signal Terminate.

It is a standard operating system signal used to request that a process terminate.

In Kubernetes, SIGTERM is normally sent to the container's main process when the container is being stopped.

Unlike SIGKILL, SIGTERM allows the application to execute shutdown logic.

SIGTERM vs. SIGKILL

Example: Handling SIGTERM in Node.js

A Node.js application can listen for SIGTERM and initiate its shutdown procedure.

const express = require("express"); const app = express(); const server = app.listen(3000, () => { console.log("Server running on port 3000"); }); process.on("SIGTERM", () => { console.log("SIGTERM received"); server.close(() => { console.log("HTTP server closed"); process.exit(0); }); }); 

The server.close() method stops the HTTP server from accepting new connections and allows existing connections to finish.

For production applications, you should also implement a bounded shutdown timeout, handle SIGINT, and close database clients, message consumers, and other resources.

This ensures that the application does not wait indefinitely for connections that never close.

4. Understanding In-Flight Request Draining

One of the most important parts of graceful shutdown is in-flight request draining.

An in-flight request is a request that has already reached the application but has not finished processing.

Consider an application processing the following requests:

Request A: Completed Request B: Processing Request C: Processing Request D: Waiting 

Now Kubernetes terminates the application pod.

Without graceful shutdown, the application might terminate while Requests B and C are still running.

This can cause failed requests and unexpected errors.

With request draining, the application follows a different process:

 SIGTERM Received | v Stop Accepting New Requests | v Mark Application Unready | v Wait for Active Requests | +-------+-------+ | | Request B Request C Complete Complete | | +-------+-------+ | v Close Connections | v Exit Process 

The application waits for ongoing requests to complete before terminating.

Why Is Request Draining Important?

Consider an API endpoint that generates a large report.

GET /api/reports/monthly 

Suppose generating the report takes 25 seconds.

If Kubernetes terminates the pod after 5 seconds, the request fails.

But if the application receives SIGTERM and waits for active requests to finish, the report can complete successfully.

This is especially important for:

• Payment processing APIs.
• File upload services.
• Report generation systems.
• Long-running database queries.
• Microservices communicating through HTTP.
• Streaming applications.
• Message processing services.

5. Readiness Probe: Preventing New Traffic During Shutdown

A readiness probe tells Kubernetes whether a container is ready to receive traffic.

When an application starts shutting down, it should become unready.

This helps Kubernetes remove the pod from Service endpoints so that new traffic is no longer routed to it.

Example readiness probe:

readinessProbe: httpGet: path: /health/ready port: 8080 initialDelaySeconds: 5 periodSeconds: 5 

The application should return:

HTTP/1.1 200 OK 

while it is ready to receive traffic.

When shutdown begins, the readiness endpoint should return:

HTTP/1.1 503 Service Unavailable 

Example Application Logic

let shuttingDown = false; app.get("/health/ready", (req, res) => { if (shuttingDown) { return res.status(503).send("Not Ready"); } res.status(200).send("Ready"); }); process.on("SIGTERM", () => { shuttingDown = true; }); 

Now, when SIGTERM is received, the application marks itself unready.

Kubernetes can then remove it from the endpoints used by Services.

Important: Readiness becoming false does not instantly guarantee that every load balancer, ingress controller, service mesh, or external client has stopped sending traffic.

Endpoint propagation and connection behavior depend on the networking architecture.

That is why applications should also handle requests safely during the draining period.

6. Kubernetes PreStop Hook Explained

Kubernetes provides lifecycle hooks that allow applications to execute specific actions during container lifecycle events.

One important lifecycle hook is preStop.

It executes immediately before Kubernetes sends SIGTERM to the container process.

Example:

apiVersion: apps/v1 kind: Deployment metadata: name: backend-api spec: replicas: 3 selector: matchLabels: app: backend-api template: metadata: labels: app: backend-api spec: terminationGracePeriodSeconds: 60 containers: - name: backend-api image: backend-api:1.0 ports: - containerPort: 8080 lifecycle: preStop: exec: command: - /bin/sh - -c - sleep 10 readinessProbe: httpGet: path: /health/ready port: 8080 periodSeconds: 5 

Why Use a PreStop Hook?

A preStop hook can help create a controlled shutdown window.

For example:

1. Kubernetes starts pod termination.
2. The preStop hook executes.
3. The application temporarily remains alive.
4. The endpoint removal and traffic propagation process gets additional time.
5. The hook completes.
6. Kubernetes sends SIGTERM.
7. The application begins its shutdown logic.

However, a fixed sleep is only a timing workaround.

A better implementation is to make the application expose a shutdown endpoint or use an application-specific drain mechanism.

For example:

lifecycle: preStop: httpGet: path: /internal/drain port: 8080 

The endpoint can mark the application as unready and initiate a controlled draining state.

The application should still handle SIGTERM and enforce its own shutdown timeout.

7. Production-Ready Kubernetes Graceful Shutdown Configuration

Let's combine the concepts into a practical Kubernetes Deployment configuration.

apiVersion: apps/v1 kind: Deployment metadata: name: production-api spec: replicas: 3 strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 maxUnavailable: 0 selector: matchLabels: app: production-api template: metadata: labels: app: production-api spec: terminationGracePeriodSeconds: 60 containers: - name: production-api image: production-api:1.0 ports: - containerPort: 8080 lifecycle: preStop: httpGet: path: /internal/drain port: 8080 readinessProbe: httpGet: path: /health/ready port: 8080 initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 2 failureThreshold: 3 livenessProbe: httpGet: path: /health/live port: 8080 initialDelaySeconds: 10 periodSeconds: 10 timeoutSeconds: 2 failureThreshold: 3 

Understanding the Important Settings

replicas: 3

Runs three application instances to maintain service availability.

maxSurge: 1

Allows Kubernetes to create one additional pod during a rolling update.

maxUnavailable: 0

Prevents the deployment controller from intentionally reducing the number of available replicas below the desired count during the rollout.

terminationGracePeriodSeconds: 60

Provides up to 60 seconds for the container to terminate.

preStop

Starts the application's drain procedure before SIGTERM is delivered.

readinessProbe

Prevents an unready application from being treated as ready for new Service traffic.

livenessProbe

Checks whether the application is alive and should continue running.

Avoid making the liveness probe fail simply because the application is shutting down. That can cause unnecessary restarts.

8. Implementing Graceful Shutdown in Spring Boot

Spring Boot applications can also implement graceful shutdown.

Spring Boot supports graceful shutdown for embedded web servers.

In application.yml:

server: shutdown: graceful spring: lifecycle: timeout-per-shutdown-phase: 45s 

What Happens?

When the application receives a shutdown signal:

1. Spring Boot stops accepting new requests.
2. Existing requests are allowed to complete.
3. The application waits for active requests within the configured timeout.
4. Spring-managed lifecycle components are stopped.
5. The application shuts down.

You can configure Kubernetes to allow sufficient time for this process:

spec: terminationGracePeriodSeconds: 60 

A 45-second Spring shutdown timeout inside a 60-second Kubernetes termination window leaves approximately 15 seconds for additional termination overhead.

The actual timing should be tested with your application, hooks, and dependencies.

9. Graceful Shutdown for Kafka Consumers and Background Workers

Graceful shutdown is not limited to HTTP applications.

It is equally important for background workers and event-driven systems.

Consider a Kafka consumer processing messages:

Kafka Topic | v Consumer Application | v Message Processing | v Database Write 

Suppose the application is terminated while processing a message.

If the consumer commits its offset incorrectly, the message could be lost from the application's processing perspective.

A reliable shutdown strategy should:

1. Stop polling for new messages.
2. Allow the current message or batch to finish.
3. Commit offsets only after successful processing.
4. Close the consumer cleanly.
5. Allow the consumer group to rebalance.

For example, with the Kafka Java client:

consumer.wakeup(); try { // Consumer polling and processing loop } catch (WakeupException e) { if (!shuttingDown) { throw e; } } finally { consumer.close(); } 

A production implementation must coordinate the shutdown flag, polling loop, processing threads, and offset commits.

For applications using Spark Structured Streaming, shutdown should also account for the streaming query lifecycle, checkpoint consistency, and the completion of ongoing micro-batches.

Do not assume that closing a Kafka consumer automatically guarantees successful processing of every in-flight message.

10. Common Graceful Shutdown Mistakes

Even applications that implement SIGTERM handling can experience downtime because of incorrect shutdown configurations.

Here are some common mistakes.

Mistake 1: Ignoring SIGTERM

If the application does not handle SIGTERM or has no shutdown procedure, it may terminate before completing active work.

Solution: Implement a controlled shutdown mechanism.

Mistake 2: Setting a Very Short Termination Grace Period

Example:

terminationGracePeriodSeconds: 5 

If the application requires 20 seconds to complete active requests, Kubernetes may forcefully terminate it.

Solution: Set the grace period based on measured shutdown duration and maximum expected in-flight work.

Mistake 3: Using Only a Sleep Command

Example:

preStop: exec: command: - /bin/sh - -c - sleep 30 

Sleeping does not stop the application from accepting new requests.

Solution: Use an application-controlled drain endpoint or shutdown mechanism.

Mistake 4: Closing Database Connections Too Early

An application might close its database connection pool immediately after receiving SIGTERM.

This can interrupt active transactions.

Solution: Wait for active requests and background operations to finish before closing database resources.

Mistake 5: Forgetting Long-Lived Connections

WebSockets, streaming responses, and persistent connections may remain open for a long time.

Solution: Implement connection draining, communicate shutdown to clients, and enforce appropriate connection deadlines.

Mistake 6: Not Testing Rolling Updates

A graceful shutdown implementation may work when manually deleting a pod but fail during actual rolling deployments.

Solution: Test real deployment scenarios and verify traffic behavior.

11. How to Test Graceful Shutdown in Kubernetes

Testing is essential before deploying graceful shutdown changes to production.

Step 1: Find Your Pod

kubectl get pods -n production 

Step 2: Monitor Pod Status

kubectl get pods -n production -w 

Step 3: Monitor Application Logs

kubectl logs -f production-api-xxxxx -n production 

Step 4: Delete a Pod

kubectl delete pod production-api-xxxxx -n production 

Step 5: Verify Shutdown Logs

You should see logs similar to:

SIGTERM received Application entering draining mode Readiness status changed to DOWN Stopping new requests Waiting for active requests Active requests completed Closing database connections Application shutdown completed 

Step 6: Test Under Load

Send continuous HTTP requests while terminating one pod.

For example:

while true do curl -s -o /dev/null \ -w "%{http_code}\n" \ http://your-service/api/health sleep 0.2 done 

Observe whether requests fail during pod termination.

For meaningful testing, use a load-testing tool and include requests that take longer to complete.

Monitor:

• HTTP 5xx error rate.
• Request latency.
• Active connections.
• Pod termination duration.
• Application shutdown logs.
• Failed or duplicated background jobs.
• Kafka consumer lag and offset behavior.

A successful test should demonstrate that new requests stop reaching the terminating pod while existing requests complete within the configured shutdown window.

12. Best Practices for Zero-Downtime Kubernetes Deployments

For production workloads, follow these practices:

A Note About PodDisruptionBudget

A PodDisruptionBudget (PDB) helps protect application availability during certain voluntary disruptions, such as node maintenance.

Example:

apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: production-api-pdb spec: minAvailable: 2 selector: matchLabels: app: production-api 

This configuration requires at least two matching pods to remain available for supported voluntary disruptions.

However, a PDB does not guarantee zero downtime and does not control every type of pod termination.

It should be used alongside proper readiness probes, rolling update settings, and graceful shutdown handling.

13. Real-World Production Scenario

Let's understand how these concepts work together.

Assume an e-commerce platform has three API pods:

 Ingress Controller | Kubernetes Service | +--------------+--------------+ | | | Pod A Pod B Pod C Running Running Running 

A new version of the application is deployed.

Kubernetes creates a replacement pod and begins terminating Pod A.

Without Graceful Shutdown

• Pod A receives termination.
• Active requests are interrupted.
• Connections are reset.
• Some customers see HTTP 502 errors.
• Background operations may fail.

With Graceful Shutdown

• Pod A enters the termination lifecycle.
• Its application drain procedure begins.
• Readiness becomes false.
• Kubernetes removes it from ready Service endpoints.
• Existing requests are allowed to complete.
• Background processing stops safely.
• Database connections close.
• Pod A exits.
• The replacement pod continues serving traffic.

The result is a much safer deployment with significantly reduced request failures.

Remember that zero downtime is an end-to-end property. Kubernetes configuration alone cannot guarantee it. The application, load balancer, networking layer, database, and clients must all behave correctly during termination.

14. Final Thoughts

Graceful shutdown is one of the most important but frequently overlooked aspects of production Kubernetes engineering.

A successful deployment is not simply about starting a new pod. It is also about terminating the old pod without disrupting ongoing operations.

The combination of SIGTERM handling, readiness probes, preStop hooks, in-flight request draining, sufficient termination grace periods, and safe resource cleanup creates a reliable shutdown strategy.

For production systems, always remember:

Do not just stop the application. Stop accepting new work, finish existing work, release resources, and then terminate.

This approach helps build resilient microservices, reliable distributed systems, and safer Kubernetes deployments.

Continue Learning with DevCampus

Understanding graceful shutdown is just one part of building reliable production systems.

If you are learning Kubernetes, DevOps, backend engineering, or distributed systems, explore more practical technical guides on DevCampus.

Here are some related topics worth exploring:

1. Kubernetes Architecture and Core Components

Understand how the Kubernetes API Server, Scheduler, Controller Manager, Kubelet, Pods, and Services work together.

Explore more: DevCampus – Kubernetes Learning Resources

2. Docker Containers and Container Lifecycle

Learn how containers work, how processes run inside containers, and how Docker handles signals and application termination.

Explore more: DevCampus – Docker and Containerization

3. Kubernetes Health Checks: Liveness, Readiness, and Startup Probes

Understand how Kubernetes determines whether an application is alive, ready to receive traffic, or still starting.

Explore more: DevCampus – Kubernetes Health Checks

4. Kubernetes Services and Networking

Learn how ClusterIP, NodePort, LoadBalancer, DNS, and service discovery route traffic between applications.

Explore more: DevCampus – Kubernetes Networking

5. Microservices Architecture and API Gateway

Understand how microservices communicate, how API gateways route requests, and how distributed applications maintain reliability.

Explore more: DevCampus – Microservices and System Design

6. Kafka Consumer Groups and Message Processing

Explore event-driven architectures, consumer groups, partition assignment, offset management, and reliable message processing.

Explore more: DevCampus – Apache Kafka Learning Resources

7. Distributed Systems: Fault Tolerance and High Availability

Understand retries, failure recovery, replication, circuit breakers, and techniques for building resilient distributed applications.

Explore more: DevCampus – Distributed Systems and System Design

Want to improve your production engineering skills?

Visit DevCampus to explore technical articles, structured learning paths, hands-on labs, and tutorials covering modern software engineering.

Learn. Build. Practice. Grow.

DevCampus – Your platform for practical software engineering knowledge.