Redis Cluster Slot Migration Tuning: Parameters, Monitoring, and Troubleshooting
Quick Answer
- What it does: The
CLUSTER MIGRATIONcommand (Redis 8.4.0+) provides atomic, online slot migration between cluster nodes with built-in write-pause and monitoring capabilities, replacing the manualCLUSTER SETSLOT+MIGRATEworkflow. - First checks: Ensure cluster state is
ok(CLUSTER INFO), target node is a master, and no existing migration tasks involve the target slots (CLUSTER MIGRATION STATUS ALL). - Minimal command:
redis-cli -h <source-node> -p <port> -c CLUSTER MIGRATION IMPORT <start-slot> <end-slot>on the destination master node. - Key tuning parameters:
cluster-slot-migration-handoff-max-lag-bytes(default 1MB) controls when write-pause triggers;cluster-slot-migration-write-pause-timeout(default 10s) limits pause duration. - Version boundary: Redis Open Source 8.4.0+ only; not compatible with Redis Software or Redis Cloud.
What Problem It Solves
Manual slot migration in Redis Cluster requires multiple steps: CLUSTER SETSLOT <slot> MIGRATING <node-id>, CLUSTER SETSLOT <slot> IMPORTING <node-id>, then iterating keys with MIGRATE. This non-atomic process risks intermediate inconsistent states, requires external scripting for progress tracking, and has no built-in write-pause mechanism.
CLUSTER MIGRATION solves these issues by providing:
- Atomic slot transfer: The entire slot range moves as a single operation.
- Built-in write-pause: Controlled by
cluster-slot-migration-handoff-max-lag-bytesandcluster-slot-migration-write-pause-timeout, minimizing write disruption during ownership handoff. - Native monitoring:
CLUSTER MIGRATION STATUSsubcommands provide detailed progress, state, and error information without external tools.
When This Error or Setup Appears
Slot migration is needed in these scenarios:
- Cluster scaling out: Adding new master nodes and redistributing slots.
- Cluster scaling in: Removing nodes after migrating their slots elsewhere.
- Load balancing: Rebalancing slot distribution across masters to equalize key counts or request load.
- Node replacement: Moving slots from a failing or underperforming node to a healthy one.
Parameters and Environment Variables
| Parameter | Required | Default | Description |
|---|---|---|---|
cluster-slot-migration-handoff-max-lag-bytes | No | 1MB | After slot snapshot completes, if remaining replication stream size falls below this threshold, the source node pauses writes to hand off slot ownership. Higher values trigger handoff earlier but may cause longer write pauses. Lower values result in shorter write pauses but may be harder to reach with steady incoming writes. |
cluster-slot-migration-write-pause-timeout | No | 10 seconds | Maximum duration the source node pauses writes during ASM handoff. If the destination fails to take over slots within this timeout, the source assumes migration failed and resumes writes. |
Set these in redis.conf or at runtime with CONFIG SET:
CONFIG SET cluster-slot-migration-handoff-max-lag-bytes 524288
CONFIG SET cluster-slot-migration-write-pause-timeout 15000
Root Cause Analysis
The migration process follows these phases:
- Snapshot: Source node takes a snapshot of the slot's keys and begins replicating them to the destination.
- Incremental sync: New writes to the source are continuously replicated to the destination.
- Handoff: When the replication lag (measured in bytes) drops below
cluster-slot-migration-handoff-max-lag-bytes, the source pauses writes, completes final replication, and transfers slot ownership to the destination. - Resume: Destination takes over the slot; source resumes normal operation without those slots.
Failures typically occur when:
- The cluster is unhealthy (not
okstate). - Network latency prevents timely replication, causing handoff timeout.
- The target node is a replica, not a master.
- Another migration task already involves the same slot.
Common Errors and Fixes
| Error | Cause | Solution |
|---|---|---|
ERR Slot 100 is already migrating from source node | Slot is already part of another migration task | Run CLUSTER MIGRATION STATUS ALL to find the task ID, then CLUSTER MIGRATION CANCEL ID <task-id> or wait for completion |
ERR Cluster state is not ok | Cluster is unhealthy | Check CLUSTER INFO for cluster_state:fail. Fix node failures or unassigned slots, then retry |
ERR Target node is not a master | Destination is a replica | Use CLUSTER NODES to verify node roles. Select a master node as target, or promote the replica first |
Task status failed with last_error containing -MOVED or -ASK | Network partition or config inconsistency during migration | Check node connectivity and cluster-require-full-coverage consistency. Cancel and restart the task during low load |
Production Notes and Security Checks
Critical limitations:
- Version requirement: Redis Open Source 8.4.0+ only. Not available in Redis Software or Redis Cloud.
- Cluster health: All slots must be assigned, cluster state must be
ok, and no concurrent migration tasks may exist for the target slots. - ACL permissions: Requires
@admin,@slow,@dangerouscategories. RestrictCLUSTER MIGRATIONto operational users only. - Write-pause risk: Brief write unavailability occurs during handoff. Set
cluster-slot-migration-write-pause-timeoutappropriately for your network conditions. - Asymmetric cancellation: Cancelling on the source node does not automatically stop the destination. You must cancel on both nodes if needed.
- Key visibility: During migration,
KEYS,SCAN, andRANDOMKEYmay filter out keys in migrating slots, creating a false impression of data loss.
Security recommendations:
- Use ACL rules to limit
CLUSTER MIGRATIONexecution to admin users. - Perform migrations during low-traffic periods.
- Monitor migration progress and errors with alerting.
- Test migration workflows thoroughly in staging environments.
FAQ
Q: How does CLUSTER MIGRATION affect running client requests?
A: The impact is transparent for most operations, with a brief write-pause window during handoff. Write requests to the migrating slot are blocked until migration completes or times out (controlled by cluster-slot-migration-write-pause-timeout). Read requests may receive -ASK redirects; clients must handle ASKING correctly. Overall, atomic migration is more predictable and less disruptive than manual multi-step migration.
Q: How do I monitor a CLUSTER MIGRATION task's progress?
A: Use CLUSTER MIGRATION STATUS ID <task-id> for detailed task info including state, retries, and timestamps. Use CLUSTER MIGRATION STATUS ALL to list all tasks. Additionally, monitor key counts with INFO KEYSPACE on the destination node and CLUSTER COUNTKEYSINSLOT <slot> on the source. Integrate with monitoring systems (e.g., Prometheus + Grafana) for continuous observation.
Q: Will data be lost if a migration task fails?
A: No. CLUSTER MIGRATION is atomic. If migration fails (network interruption, node crash), the task is marked as failed but data is preserved. Source node data remains unchanged; any partially migrated data on the destination is rolled back or discarded. Check last_error for the failure reason, fix the issue, and retry. Cancelling a task (CANCEL) also does not cause data loss—both nodes revert to pre-migration state.