Fix Docker Container Exit Code 137: Root Causes and Minimal Checks

Quick Answer

• Exit code 137 (128 + 9) means the container received a SIGKILL signal, most commonly from the OOM (Out-Of-Memory) killer or an external process termination.
• First check: run docker inspect <container> --format '{{.State.OOMKilled}}' to confirm if OOM caused the exit. If false, check dmesg | grep -i kill on the host for kernel-level kill events.
• Minimal fix: increase container memory limit with --memory flag, or set --memory-swap to the same value as --memory to disable swap and prevent swap-related OOM kills.
• For Kubernetes environments, verify Pod resource limits are not too low and check kubectl describe pod for node-level OOM events. Set QoS class to Guaranteed by making limits equal to requests.
• This guidance applies to Linux containers on Docker Engine and Kubernetes; Windows container behavior differs and is not covered here.

What Problem It Solves

Exit code 137 is one of the most common and confusing container failures in production. Unlike application-level errors (exit code 1) or segmentation faults (exit code 139), exit code 137 indicates the container was forcibly killed by the operating system or orchestrator. This document provides a systematic approach to diagnosing whether the cause is OOM, manual termination, or Kubernetes eviction, and offers concrete configuration fixes to prevent recurrence.

Root Cause Analysis

Exit code 137 equals 128 + 9, where 9 is the SIGKILL signal number. SIGKILL cannot be caught or ignored by the application. The three primary causes are:

1. OOM Killer (most common): When a container exceeds its memory limit, the Linux kernel OOM killer terminates the process. Docker reports this as exit code 137 with OOMKilled: true in docker inspect.

2. External SIGKILL: Running docker kill or docker stop (after timeout) sends SIGKILL. Kubernetes also sends SIGKILL during pod eviction or when a pod exceeds its termination grace period.

3. Kernel-level OOM: Even if docker inspect shows OOMKilled: false, the host kernel may have killed the process due to system-wide memory pressure. Check dmesg for evidence.

Minimal Working Configuration

Docker Run with Memory Limits

BASH
# Set explicit memory and swap limits to prevent OOM
docker run --memory 512m --memory-swap 512m myapp

# Lower OOM killer priority to reduce chance of being killed
docker run --memory 512m --memory-swap 512m --oom-score-adj -1000 myapp

Docker Compose (Single Node)

YAML
services:
  app:
    image: myapp
    mem_limit: 512m
    memswap_limit: 512m
    oom_score_adj: -1000

Docker Compose (Swarm Mode)

YAML
services:
  app:
    image: myapp
    deploy:
      resources:
        limits:
          memory: 512M
        reservations:
          memory: 256M

Kubernetes Pod with Guaranteed QoS

YAML
apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  containers:
  - name: app
    image: myapp
    resources:
      requests:
        memory: "512Mi"
      limits:
        memory: "512Mi"

Setting limits equal to requests gives the pod a Guaranteed QoS class, making it less likely to be evicted under memory pressure.

Common Errors and Fixes

Error: OOMKilled false but exit code 137 persists

Solution: Run dmesg | grep -i kill on the Docker host. If you see entries like Killed process 12345 (myapp) total-vm:..., the kernel OOM killer acted at the system level, not the container level. This can happen even when docker inspect reports OOMKilled: false. Consider reducing the container's memory footprint or adding more RAM to the host.

Error: Container killed after setting memory limit

Solution: By default, Docker sets --memory-swap to twice the --memory value. If the host swap is limited or disabled, the container may be killed when it tries to use swap. Always set --memory-swap explicitly:

BASH
# Disable swap entirely
docker run --memory 512m --memory-swap 512m myapp

# Or set a specific swap limit
docker run --memory 512m --memory-swap 1g myapp

Production Notes and Security Checks

• Monitor OOM events proactively: Use docker events --filter 'event=oom' to watch for OOM kills in real time. Integrate with Prometheus/Grafana using the container_oom_events_total metric from cAdvisor.
• Check cgroup version: On systems using cgroup v2 (default in modern Linux distributions), memory accounting and OOM behavior differ slightly. Verify with stat -fc %T /sys/fs/cgroup/. cgroup v2 reports OOM kills more accurately.
• Swap considerations: Disabling swap (--memory-swap equal to --memory) prevents swap-related OOM kills but may cause immediate OOM under memory pressure. For latency-sensitive applications, this is often preferred.
• Application profiling: For Java applications, use -XX:+PrintGCDetails -XX:+HeapDumpOnOutOfMemoryError to capture heap dumps. For Python, use tracemalloc or memory_profiler. For Node.js, use --max-old-space-size to limit heap.
• Docker Compose vs Swarm: The deploy.resources syntax only works in swarm mode. For single-node Compose, use mem_limit and memswap_limit (deprecated but still functional) or pass --memory via docker-compose run.

FAQ

Q: Exit code 137 and exit code 139 — what's the difference?

A: Exit code 137 (128 + 9) means the container received SIGKILL, typically from OOM killer or manual termination. Exit code 139 (128 + 11) means SIGSEGV, caused by the application accessing invalid memory (null pointer dereference, stack overflow, etc.). 137 is external forced termination; 139 is an internal program error.

Q: How to distinguish OOM from manual docker kill causing exit code 137?

A: Run docker inspect <container> --format '{{.State.OOMKilled}}'. If true, OOM caused the exit. If false, check docker events --since 5m for kill events and dmesg | grep -i kill on the host for kernel-level kills. Kubernetes users should also check kubectl describe pod for eviction events.

Q: How to set memory limits in Docker Compose to avoid exit code 137?

A: For single-node Compose, use:

YAML
services:
  app:
    image: myapp
    mem_limit: 512m
    memswap_limit: 512m

For swarm mode, use:

YAML
services:
  app:
    image: myapp
    deploy:
      resources:
        limits:
          memory: 512M
        reservations:
          memory: 256M

Note: deploy.resources is ignored in non-swarm mode. Always verify with docker-compose config that your settings are applied.