Fix Elasticsearch Cluster Yellow or Red Status: Diagnosis and Repair Steps

Quick Answer

• Conclusion: A yellow cluster status means all primary shards are assigned but some replica shards are unassigned; red means at least one primary shard is unassigned. Both indicate allocation failures that require structured diagnosis.
• First check: Run GET _cluster/health to confirm status, then GET _cat/shards to find unassigned shards. For single-node clusters, set index.number_of_replicas to 0.
• Minimal fix: Use GET _cluster/allocation/explain with {"index": "<index>", "shard": <id>, "primary": true/false} to get the exact rejection reason, then address it (disk space, node count, allocation filters).
• Applicable versions: Elasticsearch 8.x (the official source references 8.19; concepts apply to 7.x and 6.x with minor API differences).

What Problem It Solves

Elasticsearch cluster health statuses—green, yellow, and red—indicate the state of shard allocation. A yellow or red status means some shards are not assigned to any node, which can lead to data unavailability, reduced search performance, and increased risk of data loss during node failures.

This guide provides a repeatable, structured approach to diagnose and fix the root causes of yellow or red cluster status, covering:

• Single-node clusters where replicas cannot be allocated
• Node failures or network partitions
• Disk space exhaustion
• Allocation setting misconfigurations
• Replica count mismatches

Root Cause Analysis

The primary diagnostic tool is the _cluster/allocation/explain API. It returns the exact reason why a shard cannot be assigned.

Step 1: Check cluster health

BASH
GET _cluster/health

Look at the status field and unassigned_shards count.

Step 2: Identify unassigned shards

BASH
GET _cat/shards?v=true&h=index,shard,prirep,state,node&s=state

Filter for rows where state is UNASSIGNED.

Step 3: Explain allocation failure

BASH
GET _cluster/allocation/explain
{
  "index": "my-index",
  "shard": 0,
  "primary": true
}

The response includes a decisions array with decider and explanation fields that pinpoint the cause.

Common root causes:

Minimal Working Configuration

Fix for Single-Node Clusters

If you have only one data node, replica shards can never be assigned (Elasticsearch never places a replica on the same node as its primary). Set replicas to 0:

BASH
PUT my-index/_settings
{
  "index.number_of_replicas": 0
}

Fix for Disk Space Issues

1. Check disk usage per node:

BASH
GET _cat/allocation?v=true&h=node,shards,disk.percent,disk.used,disk.avail,disk.total

2. Free space by deleting indices (after backup):

BASH
DELETE old-index-name

3. Or force merge read-only indices to reduce segment count:

BASH
POST my-index/_forcemerge?max_num_segments=1

Fix for Replica Count Mismatch

If you have 2 data nodes but index.number_of_replicas is set to 3, reduce it:

BASH
PUT my-index/_settings
{
  "index.number_of_replicas": 1
}

Common Errors and Fixes

Production Notes and Security Checks

Permissions Required

The user or API key performing these operations needs:

• cluster:monitor/health — for health checks
• cluster:admin/reroute — for allocation explain and reroute
• indices:admin/settings/update — for modifying index settings
• indices:admin/forcemerge — for force merge operations

Create a dedicated role with only these permissions for automation scripts.

Critical Limitations

1. Concurrent conflicts: Running multiple instances of repair scripts simultaneously can cause configuration conflicts. Use distributed locks or ensure serial execution.
2. Network security: Never expose Elasticsearch HTTP ports (9200) directly. Use a reverse proxy (Nginx) or enable TLS and IP filtering.
3. Data loss risk: Before deleting indices or running force merge, verify you have snapshots. Force merge is an I/O-intensive operation—run during low-traffic periods.
4. Configuration persistence: Dynamic settings changed via API (e.g., index.number_of_replicas) may be lost on cluster restart. Write them to elasticsearch.yml or use persistent settings APIs.

Force Merge Caution

Force merging to max_num_segments=1 is irreversible for append-only indices. For indices that receive writes, it can cause large segment merges later. Only force merge indices that are no longer being written to.

FAQ

Q: My cluster status is yellow, but all nodes are running. Why can't replica shards be allocated?

A: The most common cause is a single-node cluster. Elasticsearch never allocates a replica shard to the same node as its primary shard. With only one data node, replicas remain unassigned forever. Set index.number_of_replicas to 0, or add more data nodes.

Q: After running _cluster/reroute, why are shards still not allocated?

A: _cluster/reroute only triggers the allocation process—it does not force immediate assignment. Elasticsearch has a default allocation delay of 1 minute to avoid unnecessary work after a node temporarily leaves. If the delay passes without allocation, use _cluster/allocation/explain to see the specific rejection reason (disk space, allocation filters, node attribute mismatches).

Q: How do I safely fix a red cluster status caused by disk space exhaustion?

A: First, identify the affected nodes with GET _cat/allocation. Then, in order of priority: 1) Delete unnecessary indices (after confirming backups exist). 2) Force merge read-only indices to reduce segment overhead. 3) Use ILM policies to migrate old indices to cheaper storage or delete them. 4) As a last resort, temporarily adjust cluster.routing.allocation.disk.watermark.low and cluster.routing.allocation.disk.watermark.high thresholds—but this is risky; add disk space or nodes as soon as possible.

Official References

• Red or Yellow Cluster Status - Elasticsearch Guide 8.19