Debug Kubernetes CrashLoopBackOff: Root Causes and Minimal Fixes

Quick Answer

• Conclusion: CrashLoopBackOff means your container starts, crashes, and Kubernetes keeps restarting it, but the container never stabilizes. The fix is to identify the underlying cause—typically OOMKilled, failed liveness probes, bad image references, or missing configuration—and correct that specific issue.
• First checks: Run kubectl describe pod <pod-name> to see events and exit codes, then kubectl logs <pod-name> --previous to read the last crashed container's output. If logs are empty, check the container's entrypoint and command.
• Minimal fix: For OOMKilled (exit code 137), increase resources.limits.memory or reduce the app's memory footprint. For liveness probe failures, adjust initialDelaySeconds and failureThreshold, or add a startupProbe for slow-starting apps.
• Environment boundary: Applies to any Kubernetes cluster (v1.20+ recommended for kubectl debug). The debugging workflow is identical across managed (EKS, GKE, AKS) and self-hosted clusters.

What Problem It Solves

CrashLoopBackOff is a Kubernetes pod status indicating that a container has repeatedly crashed after starting. The kubelet attempts to restart the container with an exponential backoff delay (10s, 20s, 40s, ... up to 5 minutes), but the container never reaches a stable running state.

This guide gives you a systematic path from symptom to root cause. Instead of guessing, you'll follow a repeatable diagnostic sequence that covers the most common failure modes:

When This Error or Setup Appears

CrashLoopBackOff appears in these common scenarios:

• New deployment: You apply a manifest and the pod enters CrashLoopBackOff immediately.
• After an update: A new image tag, changed environment variables, or modified ConfigMap causes the app to crash on startup.
• Resource pressure: The node runs out of memory, and the kernel kills your container (OOMKilled).
• Dependency unavailability: The app can't reach a database, API, or other service at startup and exits.
• Misconfigured probes: The liveness probe starts too early or points to a wrong path, so Kubernetes kills a healthy container.

Root Cause Analysis

Follow this sequence to identify the root cause. Do not skip steps—each one narrows the problem space.

Step 1: Inspect Pod Status and Events

BASH
kubectl describe pod <pod-name> -n <namespace>

Look for these fields:

• State: Waiting, Running, or Terminated
• Last State: Terminated with Reason (e.g., OOMKilled, Error) and Exit Code
• Events: Liveness probe failed, Back-off restarting failed container, FailedMount, Failed to pull image

Step 2: Read the Logs

BASH
# Current logs (may be empty if the container crashes immediately)
kubectl logs <pod-name> -n <namespace>

# Logs from the previous (crashed) container instance
kubectl logs <pod-name> -n <namespace> --previous

# Tail the last N lines
kubectl logs <pod-name> -n <namespace> --previous --tail=50

If --previous returns nothing, the container likely crashed before writing any output—check the entrypoint and command.

Step 3: Check Resource Usage

BASH
kubectl top pod <pod-name> -n <namespace>

Compare actual memory usage against the resources.limits.memory in your pod spec. If usage is at or near the limit, you're hitting OOMKilled.

Step 4: Verify Configuration Sources

BASH
kubectl get configmap <config-name> -n <namespace>
kubectl get secret <secret-name> -n <namespace>

Confirm that every ConfigMap and Secret referenced in your pod spec exists in the correct namespace. A missing ConfigMap causes a FailedMount event and the container never starts.

Step 5: Interactive Debugging

When logs are unavailable or unhelpful, use kubectl debug to run a sidecar container in the same pod:

BASH
kubectl debug <pod-name> -n <namespace> -it --image=busybox --copy-to=<debug-pod-name> --share-processes

This gives you a shell in the pod's network and filesystem context. You can inspect environment variables, mounted volumes, and test connectivity to dependencies.

Common Errors and Fixes

Error 1: OOMKilled (Exit Code 137)

Symptom: Pod status shows OOMKilled, exit code 137. The kernel killed the container because it exceeded its memory limit.

Fix: Increase the memory limit or reduce the application's memory footprint.

YAML
resources:
  requests:
    memory: "256Mi"
  limits:
    memory: "512Mi"   # Increase this value

For language-specific tuning:

• Java: Adjust -Xmx to fit within the limit, e.g., -Xmx256m
• Node.js: Set --max-old-space-size=256
• Go: Use runtime/debug.SetMemoryLimit() (Go 1.19+)

Error 2: Liveness Probe Failed

Symptom: Events show Liveness probe failed: HTTP probe failed with statuscode: 500. Kubernetes kills and restarts the container.

Fix: Give the app more startup time and verify the probe path.

YAML
livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 15   # Increase if the app is slow to start
  periodSeconds: 10
  failureThreshold: 3       # Allow more failures before restart

For applications that take a long time to become ready, add a startupProbe:

YAML
startupProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
  failureThreshold: 30      # 30 * 5s = 150s max startup time

Error 3: ImagePullBackOff

Symptom: Pod status is ImagePullBackOff. The container never starts.

Fix: Verify the image name and tag, and configure authentication for private registries.

BASH
# Check the exact image reference in your deployment
kubectl get deployment <deployment-name> -o jsonpath='{.spec.template.spec.containers[0].image}'

For private registries, create a secret and reference it:

BASH
kubectl create secret docker-registry regcred \
  --docker-server=<registry-server> \
  --docker-username=<username> \
  --docker-password=<password>

YAML
spec:
  imagePullSecrets:
    - name: regcred

Error 4: Empty Logs and Immediate Exit

Symptom: The container exits within seconds, logs are empty, and no OOM or probe events appear.

Fix: Check the container's command and args. A common mistake is overriding the image's default entrypoint incorrectly.

YAML
containers:
  - name: app
    image: myapp:1.0
    command: ["/app/start.sh"]   # Verify this path exists in the image
    args: ["--config", "/etc/app/config.yaml"]

Test the command locally:

BASH
docker run --rm myapp:1.0 /app/start.sh --config /etc/app/config.yaml

If the command fails locally, it will fail in Kubernetes.

Parameters and Environment Variables

When debugging CrashLoopBackOff, these kubectl flags and commands are the ones you'll use most:

For container configuration, these pod spec fields are the most relevant:

Production Notes and Security Checks

• RBAC permissions: Use a service account with least-privilege access for debugging. Avoid granting cluster-admin for routine troubleshooting. Create a role with get, list, and logs permissions on pods.
• API server load: kubectl describe and kubectl logs hit the API server. In large clusters, avoid running these in tight loops. Use --tail to limit log output.
• Log persistence: Container logs disappear when the pod is deleted. Configure a log aggregation system (EFK/ELK, Loki, or Datadog) to retain logs for post-mortem analysis.
• Debug container security: kubectl debug gives you a shell in the pod's context. Restrict who can use it, and audit debug sessions. Consider using ephemeral containers with kubectl debug --target to avoid modifying the original pod spec.
• Secret handling: Never put registry credentials or other secrets directly in the pod spec. Use imagePullSecrets referencing a docker-registry secret, and manage secrets through a tool like Sealed Secrets or External Secrets.
• Config drift: Manage ConfigMaps and Secrets through GitOps (Argo CD, Flux) or at minimum version-controlled manifests. Manually created configs drift from the source of truth and cause hard-to-debug startup failures.
• Network exposure: If you use kubectl port-forward for debugging, ensure you don't expose internal services unintentionally. Port-forward binds to localhost by default—keep it that way.

FAQ

Q: CrashLoopBackOff 和 ImagePullBackOff 有什么区别？

A: CrashLoopBackOff means the container starts and then crashes repeatedly—the application code, resource limits, or probes are the problem. ImagePullBackOff means Kubernetes cannot pull the container image at all, so the container never starts. ImagePullBackOff is caused by a wrong image name/tag, missing registry credentials, or network issues; CrashLoopBackOff is caused by application errors, OOMKilled, or failed health checks.

Q: 如何查看 CrashLoopBackOff 的根因？

A: Run kubectl describe pod <pod-name> to see the exit code and events. Then run kubectl logs <pod-name> --previous to read the crashed container's output. If logs are empty, check the container's command and args in the pod spec. For interactive debugging, use kubectl debug <pod-name> -it --image=busybox to get a shell in the pod's context.

Q: 如何防止 CrashLoopBackOff 在生产环境发生？

A: Test locally before deploying, including resource limits and probe behavior. Implement proper health endpoints (/health and /ready) and add a startupProbe for slow-starting applications. Set realistic resources.requests and limits based on observed usage. Verify all ConfigMaps and Secrets exist before applying the deployment. Use init containers to wait for dependent services. Monitor cluster resource usage to catch memory pressure early.