Docker Health Check Restart Unhealthy: Configure Automatic Container Recovery

Quick Answer

• Conclusion: Docker's built-in HEALTHCHECK instruction lets you automatically restart containers when the application inside becomes truly unresponsive, not just when the process crashes. This provides deeper reliability than simple restart policies.
• First checks: Verify your application exposes a health endpoint (HTTP, TCP, or script-based). Ensure curl or wget is installed inside the container if using HTTP checks. Test the health command manually with docker exec.
• Minimal fix: Add HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 CMD curl -f http://localhost:3000/health || exit 1 to your Dockerfile, then configure restart: always in Docker Compose.
• Version boundary: Docker Engine 1.12+ supports HEALTHCHECK. Docker Compose file format 2.1+ supports healthcheck and depends_on with condition: service_healthy.

What Problem It Solves

Standard Docker restart policies (restart: always, restart: unless-stopped) only restart containers when the main process exits. If your application is still running but stuck in a deadlock, leaking memory, or unable to serve requests, Docker considers it "healthy" because the process is alive.

HEALTHCHECK solves this by periodically running a command inside the container. If the command fails repeatedly, Docker marks the container as unhealthy. Combined with a restart policy, this triggers an automatic restart—recovering the service without manual intervention.

This is especially critical in:

• Microservices architectures where each service must be independently recoverable
• Production deployments where uptime matters
• Systems that need self-healing capabilities without human operators

Minimal Working Configuration

Dockerfile Approach

DOCKERFILE
FROM node:18-alpine

WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .

# Install curl for health checks
RUN apk add --no-cache curl

# Expose health endpoint in your app (example Express route)
# app.get('/health', (req, res) => res.send('OK'))

HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
  CMD curl -f http://localhost:3000/health || exit 1

EXPOSE 3000
CMD ["node", "server.js"]

Docker Compose Approach

YAML
version: '3.8'

services:
  web:
    build: .
    ports:
      - "3000:3000"
    restart: always
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s

  database:
    image: postgres:15
    environment:
      POSTGRES_USER: user
      POSTGRES_PASSWORD: pass
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U user"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 30s

  app:
    build: ./app
    depends_on:
      database:
        condition: service_healthy
    restart: always

Parameters and Environment Variables

Important notes:

• All values are specified as durations (e.g., 30s, 2m, 1h30m)
• The test command can be a shell command (CMD curl ...) or an exec array (["CMD", "curl", "-f", "http://localhost/health"])
• Using CMD-SHELL format allows environment variable expansion: ["CMD-SHELL", "curl -f http://$HOST:3000/health"]

Root Cause Analysis

When a container is marked as unhealthy and restarts, the typical flow is:

1. Docker runs the health check command at each interval
2. If the command exits with non-zero or times out, it counts as a failure
3. After retries consecutive failures, the container status changes from healthy to unhealthy
4. With restart: always, Docker stops and restarts the unhealthy container
5. After restart, the start_period grace period begins again

Common root causes for health check failures:

• Application deadlock or infinite loop (process alive but not responding)
• Memory exhaustion causing slow responses
• Database connection pool exhaustion
• File descriptor limits reached
• External dependency timeout (if using deep health checks)

Common Errors and Fixes

Error: Health check command returned non-zero exit code immediately

Health check command returned non-zero exit code (1) immediately, container marked as unhealthy.

Solution: The health check command itself is failing. Debug by running it manually:

BASH
docker exec -it <container_id> sh -c "curl -f http://localhost:3000/health"

Common causes:

• Application hasn't started listening on the expected port
• Wrong port or path in the health check URL
• curl or wget not installed in the container
• Firewall or network namespace issues

Error: Health check timed out

Health check timed out after 30s, container marked as unhealthy.

Solution: The check takes longer than the --timeout setting. Either:

• Optimize your application's health endpoint response time
• Increase --timeout (e.g., --timeout=60s)
• Check if your custom script has long-running operations

Error: Container restarting repeatedly

Container restarting due to consecutive health check failures.

Solution: The application keeps becoming unhealthy. Check application logs for:

• Memory leaks (monitor with docker stats)
• Database connection issues
• File descriptor exhaustion

Consider increasing --interval and --retries to avoid rapid restart cycles, or use a shallower health check that only verifies the process is responsive.

Error: depends_on condition service_healthy never met

depends_on condition service_healthy never met, service stuck in starting state.

Solution: The dependency service's health check never passes. Debug by:

1. Checking the dependency service logs
2. Running its health check command manually
3. Temporarily removing condition: service_healthy to isolate the issue

For PostgreSQL, ensure pg_isready uses the correct user: pg_isready -U <username>

Production Notes and Security Checks

Critical Limitations

1. Cascading failures: If you use deep health checks (checking database connectivity) as the restart trigger, a brief database outage can cause all dependent services to restart simultaneously. Use shallow checks for restart decisions and deep checks for load balancer routing.

2. Startup storms: An incorrect start_period can cause containers to restart in an infinite loop during initialization. Always set start_period to at least 1.5x your application's expected startup time.

3. Resource consumption: Frequent health checks (interval < 10s) or complex checks (database queries every check) consume CPU and network resources. Balance check frequency with your reliability requirements.

4. File locking: If your health check script writes to files (logs, temp files), container restarts can cause file locking issues. Use in-memory checks when possible.

Security Considerations

• Health check commands run inside the container with the same user as the main process
• Avoid exposing sensitive information in health check endpoints
• For production, consider rate-limiting health check endpoints to prevent abuse
• Use CMD-SHELL format carefully to avoid shell injection if the command includes user input

Best Practices

• Shallow checks for restart: Only check if the process is responsive (HTTP 200 or TCP connection)
• Deep checks for routing: Check database, cache, and external dependencies for load balancer health probes
• Set start_period generously: Account for initialization time, database migrations, and warm-up
• Monitor unhealthy events: Set up alerts when containers become unhealthy to detect systemic issues
• Use custom scripts for complex logic: Shell scripts can check multiple conditions, disk space, or external APIs

FAQ

Q: My application takes 2 minutes to start, but health checks begin after 30 seconds and cause restarts. What should I do?

A: Use the --start-period parameter. Set start_period: 120s in your Docker Compose or --start-period=120s in your Dockerfile. During this period, health check failures don't count toward retries, giving your application time to fully initialize.

Q: Should I use shallow or deep health checks for restart decisions?

A: For Docker's native HEALTHCHECK, use shallow checks (process responsiveness only). Deep checks (database, external APIs) are better suited for load balancer traffic routing. Using deep checks as restart triggers can cause cascading failures when a shared dependency becomes temporarily unavailable.

Q: My health check script needs environment variables, but they're not available inside the container. How do I fix this?

A: The issue is likely the command format. In Docker Compose, using ["CMD", "curl", "-f", "http://$HOST:3000/health"] (JSON array) doesn't invoke a shell, so environment variables aren't expanded. Use ["CMD-SHELL", "curl -f http://$HOST:3000/health"] instead, or hardcode the values in your script.