Kafka 4.0+ New Consumer Rebalance Protocol: Migration and Configuration Guide

Quick Answer

• What to do: To use the new incremental consumer rebalance protocol in Apache Kafka 4.0+, set group.protocol=consumer in your client configuration and configure server-side heartbeat/timeout settings via group.consumer.heartbeat.interval.ms and group.consumer.session.timeout.ms.
• First checks: Verify your Kafka cluster runs 4.0 or later; ensure the consumer group is empty before switching protocols; confirm your custom partition assignors are migrated to server-side implementations.
• Minimal client config: Add group.protocol=consumer to your consumer properties. No other client-side rebalance parameters are needed—heartbeat and session timeout are now server-controlled.
• Key benefit: Eliminates global sync barriers during rebalancing, reducing rebalance time from seconds to milliseconds in large consumer groups.
• Version boundary: This protocol is available only in Apache Kafka 4.0+. Classic protocol remains the default for backward compatibility.

What Problem It Solves

The classic consumer rebalance protocol (pre-4.0) uses a stop-the-world approach: when a consumer joins or leaves a group, all consumers must synchronously revoke partitions, rejoin, and receive new assignments. This causes significant latency spikes in large consumer groups or environments with frequent scaling events.

The new Consumer protocol introduces an incremental rebalance design:

• No global synchronization barrier
• Partitions are reassigned incrementally as consumers join or leave
• Heartbeat and session timeout management moves from client to server, reducing misconfiguration risks
• Supports online rolling upgrades and downgrades without cluster downtime

This protocol is ideal for high-throughput streaming applications, event-driven microservices, and real-time log aggregation where consumer count fluctuates due to auto-scaling.

Parameters and Environment Variables

Server-Side Configuration (broker properties)

Client-Side Configuration (consumer properties)

Important: When using group.protocol=consumer, the client-side heartbeat.interval.ms and session.timeout.ms are ignored. Configure these on the broker instead.

Root Cause Analysis

The classic protocol's rebalance latency stems from its synchronous design:

1. A coordinator detects a group membership change
2. It sends a LeaveGroup or JoinGroup request to all members
3. All consumers must stop processing, revoke partitions, and rejoin
4. The coordinator runs the assignment algorithm and distributes results
5. Only after all consumers acknowledge can processing resume

This creates a global synchronization barrier where the slowest consumer determines rebalance duration. In groups with hundreds of consumers or high partition counts, this can take tens of seconds.

The new Consumer protocol replaces this with an incremental, asynchronous model:

• The coordinator tracks partition ownership per consumer
• When a consumer joins/leaves, only the affected partitions are reassigned
• No global barrier—other consumers continue processing unaffected partitions
• Heartbeat monitoring moves to the server, eliminating client-side timing bugs

Common Errors and Fixes

Error: "Consumer group 'my-group' is not empty and cannot be converted to 'Consumer' protocol."

Cause: You attempted to switch group.protocol while consumers are still running with the classic protocol.

Fix: Stop all consumers in the group, set group.protocol=consumer, then restart them. The group must be empty before protocol conversion.

BASH
# Stop all consumers (example with kafka-consumer-groups)
kafka-consumer-groups --bootstrap-server localhost:9092 --group my-group --reset-offsets --to-earliest --execute
# Then restart consumers with group.protocol=consumer

Error: "Unsupported assignor 'custom-assignor' for Consumer group protocol."

Cause: Your client configuration specifies a custom partition assignor that exists only on the client side.

Fix: Migrate your custom assignor to a server-side implementation. Implement org.apache.kafka.server.group.share.ConsumerGroupPartitionAssignor, package it, add to broker classpath, and configure group.consumer.assignors on the broker.

PROPERTIES
# broker.properties
group.consumer.assignors=com.example.MyCustomAssignor,uniform

Error: "Rebalance failed due to incompatible assignor metadata in Classic group."

Cause: During an online rolling upgrade, the classic group uses an assignor that embeds custom metadata (e.g., a custom StickyAssignor variant).

Fix: Either perform an offline upgrade (stop all consumers, switch protocol, restart) or modify the classic group's assignor to one that doesn't embed custom metadata (e.g., standard RangeAssignor or RoundRobinAssignor).

Error: "Heartbeat interval or session timeout configuration ignored when using Consumer protocol."

Cause: You set heartbeat.interval.ms or session.timeout.ms in the client configuration, but these are server-controlled under the new protocol.

Fix: Remove these client-side settings and configure them on the broker:

PROPERTIES
# broker.properties
group.consumer.heartbeat.interval.ms=5000
group.consumer.session.timeout.ms=60000

Production Notes and Security Checks

Limitations

• Kafka 4.0+ only: The new protocol is not available on older clusters.
• No client-side custom assignors: All partition assignment logic must run on the server. Migrate any custom PartitionAssignor implementations.
• Rack-aware assignment: Not fully supported yet (tracked as KAFKA-19387).
• Online upgrade constraints: Classic groups using assignors with embedded custom metadata must upgrade offline.
• Downgrade delay: When rolling back to classic protocol, all new-protocol consumers must leave the group before conversion completes, causing brief unavailability.

Security Recommendations

1. Enable TLS/SSL encryption for all broker-client communication.
2. Use SASL authentication (PLAIN, SCRAM, or GSSAPI) for consumer connections.
3. Restrict assignor classes in group.consumer.assignors to only trusted implementations to prevent arbitrary code execution on brokers.
4. Monitor group.version changes via audit logs to detect unauthorized protocol switching.

Monitoring

New protocol introduces these JMX metrics (under kafka.consumer:type=consumer-coordinator-metrics):

Export these via Prometheus + JMX Exporter or your preferred monitoring stack.

FAQ

Q: How do I upgrade from Classic to Consumer protocol without affecting production traffic?

A: Perform an online rolling upgrade: 1) Ensure the classic group uses a standard assignor (RangeAssignor, RoundRobinAssignor) without embedded custom metadata; 2) Restart each consumer instance one by one, adding group.protocol=consumer to its configuration; 3) The first new-protocol consumer triggers automatic group conversion, maintaining interoperability with remaining classic consumers; 4) After all consumers upgrade, the group fully uses the new protocol. Note: one extra rebalance may occur during the transition.

Q: How do I implement a custom partition assignment strategy under the new protocol?

A: Implement the server-side interface org.apache.kafka.server.group.share.ConsumerGroupPartitionAssignor. Package your implementation as a JAR, add it to the broker's classpath, and configure group.consumer.assignors in broker properties (e.g., group.consumer.assignors=com.example.MyAssignor,uniform). Clients can select a specific assignor via group.remote.assignor. Client-side custom assignors are not supported.

Q: What thread model improvements does the new protocol bring, and how do I monitor them?

A: The new protocol separates heartbeat and rebalance logic from the user's main processing thread, reducing the risk of user code blocking coordinator operations. Monitor these JMX metrics: consumer.coordinator.heartbeat.rate (heartbeat frequency), consumer.coordinator.heartbeat.response.time.max (response latency), consumer.coordinator.rebalance.latency.avg (rebalance duration), and consumer.coordinator.rebalance.total (rebalance count). These are accessible via JMX or monitoring tools like Prometheus with JMX Exporter.