Fix: CrashLoopBackOff in Kubernetes

kubectl get pods shows a growing restart count and this status:

NAME                   READY   STATUS             RESTARTS      AGE
api-5d8f7c6b9d-qm4tz   0/1     CrashLoopBackOff   6 (82s ago)   9m

and kubectl describe pod lists events like:

Warning  BackOff  kubelet  Back-off restarting failed container api in pod api-5d8f7c6b9d-qm4tz_shop(...)

CrashLoopBackOff is not an error by itself. It means the container keeps exiting, and the kubelet is waiting before restarting it again. The delay doubles each time (10s, 20s, 40s and so on) up to five minutes, and resets after the container has run for ten minutes without problems. The real cause is whatever makes the process exit.

Step 1: read the logs of the crashed instance

kubectl logs api-5d8f7c6b9d-qm4tz -n shop --previous
kubectl logs api-5d8f7c6b9d-qm4tz -n shop -c api --previous

--previous shows the output of the container that just died. Without it you often see an empty log from the new container that has not started yet. Look at the last lines: a stack trace, “missing environment variable”, “connection refused” or “permission denied” usually names the cause.

Step 2: read the exit code and reason

kubectl describe pod api-5d8f7c6b9d-qm4tz -n shop

Under Last State: Terminated, check Reason and Exit Code:

• Reason: Error with exit code 1 (or another small number): the application exited on its own. The logs explain why.
• Reason: OOMKilled, exit code 137: the container went over its memory limit. See the OOMKilled fix.
• Exit code 137 without OOMKilled: the process was killed with SIGKILL, often after a failed liveness probe.
• Exit code 0 with Reason: Completed: the process finished normally, but a Deployment expects it to run forever. The command is wrong, or the work belongs in a Job.
• Exit code 127 or 126: the command was not found or not executable. Check command/args and the image’s entrypoint.

Common causes and fixes

1. Bad configuration: a missing environment variable, a wrong ConfigMap key, a wrong database URL. Fix the ConfigMap or Secret, then kubectl rollout restart deployment/api -n shop, because environment variables are only read at start.
2. A dependency is not reachable: the app exits when the database is down. Make it retry instead of exiting, or check the dependency with a readiness probe.
3. Liveness probe too strict: the event list shows Liveness probe failed and then Killing. The app needs more time to start. Add a startupProbe or raise failureThreshold and periodSeconds; do not make liveness depend on other services.
4. Out of memory: raise the memory limit or fix the leak.
5. Permissions: the image expects to run as root or write to a read-only filesystem, but the Pod’s securityContext forbids it. The log typically says “permission denied”.
6. Wrong command: command in the manifest overrides the image ENTRYPOINT and args overrides CMD; a typo there exits immediately.

Step 3: inspect the container interactively

If the container dies too fast to look inside, start a copy of the pod with a shell instead of the normal command, investigate, then delete the copy:

kubectl debug api-5d8f7c6b9d-qm4tz -n shop -it --copy-to=api-debug --container=api -- sh
kubectl delete pod api-debug -n shop

This works only if the image contains a shell. For a minimal image, use kubectl debug with a separate --image=busybox:1.36 and --share-processes on the copy.

Step 4: confirm the fix

kubectl rollout status deployment/api -n shop
kubectl get pods -n shop -l app=api -w

The restart count stops growing and the pod stays Running with READY 1/1. If a new release introduced the crash, kubectl rollout undo deployment/api -n shop restores the previous version while you investigate.

Official documentation

• Pod Lifecycle
• Debug Running Pods
• Determine the Reason for Pod Failure