Fix Kubernetes CrashLoopBackOff in k3s: a practical debug guide

Your pod keeps crashing. Kubernetes restarts it, it crashes again, Kubernetes restarts it again. kubectl get pods shows CrashLoopBackOff, and the restart count keeps climbing. I have seen this more times than I can count on my k3s clusters, and every time the fix starts the same way: stop guessing, read the logs, and follow the debug flow.

In this post I will show you how I debug a CrashLoopBackOff pod in k3s, step by step, using kubectl logs, kubectl describe, and kubectl get events. Then I will walk through the five causes I see most often, and how to fix each one.

What CrashLoopBackOff actually means

CrashLoopBackOff is not an error in your application. It is Kubernetes telling you that your container starts, exits with an error, gets restarted, and exits again — in a loop. After each crash, Kubernetes waits a little longer before restarting it. That waiting time is the "back-off" part: 10 seconds, then 20, then 40, up to 5 minutes.

The pod is not stuck. It is doing exactly what it was told to do: restart a container that keeps failing. Your job is to find out why the container fails.

First, see the problem with your own eyes:

kubectl get pods 

You will see something like this:

NAME READY STATUS RESTARTS AGE my-app-7d8f6b4c5-x9k2p 0/1 CrashLoopBackOff 12 8m 

0/1 means the container is not ready. RESTARTS 12 in 8 minutes means it has crashed 12 times. Now you debug.

Step 1: read the logs

Always start with the logs. Most of the time, the answer is right there.

kubectl logs my-app-7d8f6b4c5-x9k2p 

If the pod has already restarted, the current container may have no logs yet. Then read the logs of the previous, crashed container:

kubectl logs my-app-7d8f6b4c5-x9k2p --previous 

That --previous flag is the one people forget. The container that crashed is the one that has the error message. The new container has only just started, and it has not failed yet.

Look for the last lines before the crash. A Python traceback, a "connection refused", a "permission denied", a "file not found", a Java stack trace — the cause is usually in the last 10 lines. If the logs are empty, that is also information: the process may be crashing before it can log anything, or it may be writing logs to a file inside the container instead of stdout. In that case, step 2 will help.

If you run more than one container in a pod, name the container:

kubectl logs my-app-7d8f6b4c5-x9k2p -c my-app --previous 

Step 2: describe the pod

If the logs do not explain the crash, describe the pod. This shows you the pod's events, its restart history, and the last state of the container — including the exit code and the reason Kubernetes recorded.

kubectl describe pod my-app-7d8f6b4c5-x9k2p 

Read the Events section at the bottom first. It is a timeline of what happened to the pod. Then look at the State and Last State of the container. Two fields matter most:

  • Exit Code: 0 means the process exited cleanly — your container's main process finished and had nothing more to do. 1 is a general application error. 137 means the container was killed, usually out of memory (OOMKilled). 139 is a segmentation fault.
  • Reason: Error, OOMKilled, Completed, or CrashLoopBackOff itself.

An exit code of 137 with reason OOMKilled tells you the whole story: the container used more memory than its limit allowed, and Kubernetes killed it. The fix is a memory limit change, not a code change. I will come back to this in the causes section.

You can read the official debugging reference in the Kubernetes documentation on debugging pods.

Step 3: check the events

Events are the cluster's own log of what it did to your pod. Sometimes the pod's own logs are clean, but the events show the real problem — a failed image pull, a failed probe, a node problem.

kubectl get events --sort-by=.lastTimestamp | grep my-app 

Or watch events live while the pod crashes:

kubectl get events --watch 

Look for Failed, Unhealthy, BackOff, and Killing events. A Liveness probe failed event means your health check is killing a healthy container — or a container that is too slow to start. A FailedMount event means a volume did not attach, and the container may be crashing because a file it needs is missing.

The five causes I see most often

Most CrashLoopBackOff cases I debug come down to one of these five. Check them in this order.

1. The application crashes on start

This is the most common cause. The app starts, hits an error, and exits. The error is almost always one of these:

  • A missing environment variable or config value. The app reads DATABASE_URL, finds nothing, and exits.
  • A missing file or a wrong path. The app expects /app/config.yaml, the file is at /config/config.yaml, and it crashes.
  • A database or service that is not reachable. The app tries to connect, gets "connection refused", retries a few times, and gives up.
  • A port that is already in use, or a permission problem on the port. Binding to port 80 as a non-root user fails.

The fix is in the logs. Read the last lines, find the error, and fix the config, the environment variable, or the service it depends on. Then check that the app starts locally with the same config before you redeploy. I test with Docker first:

docker run --rm -e DATABASE_URL="postgres://user:pass@db:5432/app" my-app:latest 

If it crashes in Docker, it will crash in Kubernetes too. Fix it there, where the feedback is fast.

2. The wrong command or entrypoint

Sometimes the image is fine, but the command Kubernetes runs is wrong. A typo in the command or args field, a script that does not exist in the image, or a shell that is not installed in the image — any of these makes the container exit immediately.

Check what the pod is actually running:

kubectl get pod my-app-7d8f6b4c5-x9k2p -o yaml | grep -A 5 "command:" 

Then check that the command exists in the image:

docker run --rm --entrypoint sh my-app:latest -c "ls -la /app/start.sh" 

If you override the entrypoint in your deployment, make sure the override matches what is actually in the image. I have seen a deployment override a working entrypoint with a script that was never copied into the image. The container started, the shell said "file not found", and the pod went into CrashLoopBackOff.

3. A failing liveness probe

A liveness probe tells Kubernetes how to check if your container is alive. If the probe fails, Kubernetes restarts the container. If your probe is wrong, Kubernetes restarts a healthy container in a loop — and the status looks exactly like a crash.

Common probe mistakes:

  • The probe checks a path that does not exist, like /healthz when the app only has /.
  • The probe timeout is too short. The app needs 10 seconds to start, the probe gives it 1 second.
  • The probe runs before the app is ready. There is no initialDelaySeconds, so the first probe hits a container that is still starting up.
  • The probe uses HTTP on a port the app does not listen on.

A good liveness probe looks like this:

livenessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 15 periodSeconds: 10 timeoutSeconds: 5 failureThreshold: 3 

The initialDelaySeconds is the important part. Give the app time to start before you start judging it. And keep the liveness probe simple — it should only check that the process is alive, not that the database is up. That is what a readiness probe is for. If the database is down, you do not want Kubernetes restarting your app in a loop. You want the pod to stay up but stop receiving traffic.

4. Resource limits: OOMKilled

If the container uses more memory than its limit, Kubernetes kills it with exit code 137 and reason OOMKilled. The pod restarts, uses too much memory again, and gets killed again. Classic CrashLoopBackOff.

Check the memory usage and the limit:

kubectl top pod my-app-7d8f6b4c5-x9k2p kubectl describe pod my-app-7d8f6b4c5-x9k2p | grep -A 3 "Limits:" 

If the limit is lower than what the app needs, raise it. But first check whether the app has a memory leak — a limit increase hides the leak for a while, and then it comes back. Look at the memory trend over a few restarts. If it climbs steadily and then crashes, it is a leak. If it jumps to the limit immediately on start, the limit is just too low.

Set requests and limits that match reality:

resources: requests: memory: "256Mi" cpu: "250m" limits: memory: "512Mi" cpu: "500m" 

Requests are what the scheduler uses to place the pod. Limits are what Kubernetes enforces. A limit that is too low kills your container. A request that is too low means your pod lands on a node that cannot give it what it needs.

5. A bad image

A bad image usually shows up as ImagePullBackOff, not CrashLoopBackOff — Kubernetes cannot download the image at all. But a wrong image can still cause a crash loop: the image tag points to a different application than you expected, the image is for the wrong architecture (ARM image on an AMD64 node), or the image was built without the file your entrypoint needs.

Check the image and the node:

kubectl describe pod my-app-7d8f6b4c5-x9k2p | grep -i "image:" kubectl get nodes -o wide 

If the image tag is latest, stop using it. latest moves, and you cannot tell which version is actually running. Pin the tag to a version, and make sure the image you built is the image you deployed. I have debugged a crash loop that turned out to be a deployment pointing at an old image with a bug that was already fixed in the new one.

Kubernetes CrashLoopBackOff debug flow: check pod status, read logs, describe the pod, check events, then fix the cause

The debug flow I follow every time

Here is the whole flow, in order. I follow it every time, and it has never let me down:

  1. kubectl get pods — confirm the status and restart count.
  2. kubectl logs <pod> --previous — read the last lines of the crashed container.
  3. kubectl describe pod <pod> — check the exit code, reason, and events.
  4. kubectl get events — look for probe failures, mount failures, and kills.
  5. Match what you found to one of the five causes above, and fix that one thing.
  6. Redeploy, watch the pod come up, and check that the restart count stays at zero.

Do not skip steps. The number of times I have seen someone change the memory limit when the real problem was a missing environment variable is too high. Read first, then fix.

How I keep this from happening in production

Debugging is reactive. The better work is making crash loops rare. Here is what I do on the clusters I run.

Test the image before it reaches the cluster. Run the container locally with the same environment variables and the same command. If it crashes locally, it will crash in Kubernetes.

Use readiness probes, not just liveness probes. A readiness probe stops traffic to a pod that is not ready. A liveness probe restarts a pod that is dead. You need both, and they should check different things.

Set resource limits from real measurements, not guesses. Run the app under load, watch kubectl top, and set the limits above what you measured — with room for a spike.

Keep the logs where you can read them. Send container logs to a central place. When a pod crashes at 3 AM, you should be able to read the logs without hunting through a node. I run GenieACS on k3s for an ISP with almost 30,000 devices, and the same rule holds there: if you cannot see what happened, you cannot fix it. You can read how I handle that scale in my GenieACS on k3s article.

Pin image tags. latest is not a version. Pin the tag, and you will always know what is running.

The same thinking applies to the rest of my infrastructure work. Keep state outside the thing that can be deleted — I wrote about that for Docker in my Docker Volumes article — and keep your containers observable, so a crash is a log line you can read, not a mystery.

You can read more about k3s itself in the k3s documentation.

FAQ: Kubernetes CrashLoopBackOff

What is CrashLoopBackOff in Kubernetes?
It is a pod status that means a container starts, exits with an error, and gets restarted by Kubernetes in a loop. After each crash, Kubernetes waits longer before the next restart. The status tells you the container keeps failing, not what is failing — you have to read the logs to find the cause.

How do I see the logs of a crashed container?
Use kubectl logs <pod-name> --previous. The --previous flag reads the logs of the last crashed container, which is the one that holds the error message. Without it, you only see the logs of the current, still-running container.

What does exit code 137 mean?
Exit code 137 usually means the container was killed because it used more memory than its limit allowed — OOMKilled. Check the memory limit with kubectl describe pod and the usage with kubectl top pod. Raise the limit, or fix a memory leak if the usage climbs steadily.

What is the difference between CrashLoopBackOff and ImagePullBackOff?
ImagePullBackOff means Kubernetes cannot download the container image at all — wrong tag, wrong registry, or no access. CrashLoopBackOff means the image was downloaded and the container started, but the process inside keeps exiting with an error.

Can a liveness probe cause CrashLoopBackOff?
Yes. If the probe checks a wrong path, times out too fast, or runs before the app has finished starting, Kubernetes restarts a container that is actually fine. Set an initialDelaySeconds that gives the app time to start, and keep the probe simple.

How do I stop a pod from restarting in a loop?
Find the cause first with logs, describe, and events, then fix that one thing — a missing environment variable, a wrong command, a bad probe, a low memory limit, or a bad image. Fixing the cause stops the loop. Deleting the pod only restarts it.

Final thoughts

A pod in CrashLoopBackOff is not a mystery. It is a container telling you, over and over, that something is wrong — and the logs, the describe output, and the events are telling you exactly what.

Read the logs first. Check the exit code. Look at the events. Then fix the one thing that is actually broken. That order matters more than any single command.

Try it on your next crash loop. Follow the flow, find the cause, and fix it. The next time a pod starts looping, you will know exactly where to look.

If this helped, let me know in the comments. What was the strangest reason a pod of yours kept crashing?

More from me: my DEV profile.