Fix Kubernetes OOMKilled Errors: Memory Limit Diagnosis and Automated Fix

Quick Answer

• Root cause: OOMKilled occurs when a container exceeds its memory limit (spec.containers[].resources.limits.memory) and the Linux kernel OOM killer terminates it. The fix is either increasing the limit or reducing the application's memory footprint.
• First checks: Run kubectl describe pod <pod-name> and check lastState.terminated.reason for OOMKilled. Then query Prometheus with container_memory_working_set_bytes to see memory usage patterns over the last 72 hours.
• Minimal fix command: kubectl patch deployment <deployment-name> -p '{"spec":{"template":{"spec":{"containers":[{"name":"<container-name>","resources":{"limits":{"memory":"<new-limit>"}}}]}}}}' — but only after analyzing historical p95/p99 metrics to determine the correct value.
• Environment boundary: Requires Kubernetes 1.10+ (for resource limits), Prometheus with container_memory_working_set_bytes metric available, and kubectl access with get and patch permissions on the target namespace.

What Problem It Solves

Kubernetes OOMKilled errors cause pod restarts, service disruptions, and degraded application availability. Manually diagnosing the root cause—whether it's a memory leak, traffic spike, or undersized limit—is time-consuming and error-prone. This approach automates the analysis by:

• Querying Prometheus for historical memory usage metrics (working set bytes)
• Calculating statistically optimal memory limits based on p95/p99 percentiles
• Distinguishing between memory leaks, load spikes, and configuration insufficiency
• Generating executable kubectl commands to apply the fix

Root Cause Analysis

OOMKilled is a termination reason set by the Linux kernel when a container's memory usage exceeds its limits.memory value. The kernel's OOM killer selects and kills the offending process to protect the node. Common root causes:

The automated analyzer uses Prometheus time-series data to classify the root cause within 30 seconds, then recommends a limit based on historical percentiles plus a safety margin.

Minimal Working Configuration

To use the analyzer, you need a Prometheus server running in your cluster and a configuration file or CLI arguments. Below is the minimal MCP server configuration for the analyzer tool:

JSON
{
  "mcpServers": {
    "kubernetes-oomkilled-analyzer": {
      "command": "python",
      "args": [
        "-m",
        "kubernetes_oomkilled_analyzer",
        "--prometheus-url",
        "http://prometheus-server.monitoring.svc.cluster.local:9090",
        "--namespace",
        "production",
        "--pod-name",
        "my-app-*",
        "--lookback-hours",
        "72"
      ]
    }
  }
}

Important: Replace the Prometheus URL with your actual in-cluster service endpoint. The default http://prometheus-server.monitoring.svc.cluster.local:9090 is a placeholder—verify your Prometheus service name and namespace.

Parameters and Environment Variables

The analyzer accepts the following CLI arguments:

The --limits and --requests parameters are critical for accurate analysis. If omitted, the tool attempts to read them from the running pod's resource spec via kubectl. Always verify these values match your actual deployment configuration.

Common Errors and Fixes

Q: Prometheus query times out or returns empty data

A: Check that the Prometheus URL is reachable from the analyzer's execution environment. Verify the pod name pattern matches (wildcards are supported). Increase --lookback-hours to capture more data. Confirm that the metric container_memory_working_set_bytes exists in your Prometheus instance—if not, you may need to enable cAdvisor metrics or use an alternative metric like container_memory_usage_bytes.

Q: kubectl command fails with permission denied

A: The ServiceAccount or user running the analyzer needs get and patch permissions on pods in the target namespace. Create a dedicated RBAC role:

YAML
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: production
  name: oom-analyzer
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "patch"]

Then bind it to the analyzer's ServiceAccount with a RoleBinding.

Q: False positive memory leak detection (normal load spikes flagged as leaks)

A: Increase the analysis window to 7 days (--lookback-hours 168) and enable p95/p99 percentile analysis. For periodic batch jobs, configure exclusion windows. The analyzer should distinguish between sustained linear growth (leak) and periodic spikes (normal load) by checking if memory returns to baseline between spikes.

Q: Pod still OOMKilled after applying the recommended limit

A: Verify the new limit was applied correctly with kubectl describe pod <pod-name>. If the limit is correct but the pod still crashes, the application likely has a genuine memory leak that requires code-level fixes (heap dump, profiler, GC tuning). The analyzer's recommendation is based on historical data—if the leak accelerates, the limit may need to be recalculated with more recent data.

Production Notes and Security Checks

Critical limitations:

1. Prometheus dependency: The analyzer requires a running Prometheus instance with container_memory_working_set_bytes metrics. Without it, no analysis is possible.
2. RBAC constraints: Requires kubectl access with get and patch permissions. In restricted environments, you may need to run it from a dedicated service account.
3. Cost risk: Automatically increasing memory limits can raise cloud costs. Always set a maximum safety margin (e.g., never exceed 2x the current limit without manual approval).
4. Burst vs leak ambiguity: Transient traffic spikes may be misclassified as memory leaks. Use longer lookback windows and percentile-based analysis to reduce false positives.
5. Language-specific leaks: The analyzer cannot detect goroutine leaks in Go, thread leaks in Java, or reference cycles in Python. These require language-specific profilers.
6. Concurrent analysis conflicts: When multiple pods OOMKilled simultaneously, the analyzer may produce conflicting recommendations. Analyze one pod at a time or implement locking.

Security recommendations:

• Restrict kubectl patch to only modify resource limits, not other pod spec fields
• Set Prometheus query timeouts to prevent resource exhaustion
• Audit all auto-generated configuration changes with a change management system
• Run the analyzer in a namespace with network policies limiting egress to only the Prometheus service