Kafka consumer group rebalancing during a rolling deploy: eager, cooperative and static membership
A Kafka consumer group splits a topic’s partitions among its members, and whenever a member joins or leaves, the group rebalances. With the classic eager protocol every member gives up all its partitions until the new assignment is agreed, so a rolling deploy of the consumers stops consumption again and again. Cooperative rebalancing pauses only the partitions that move, and static membership avoids rebalancing at all for a quick restart. With stateful consumers, every moved partition also has to rebuild its state before it can continue.
The playground restarts six consumers of 12 partitions one minute apart and tracks the total lag. Each partition takes 100 messages a second, and consumers can go 50% faster than that to catch up.
A rolling deploy restarts 6 consumers sharing 12 partitions, one a minute, each gone for 20 s; every rebalance takes 5 s. Eager rebalancing stops every partition twice per restart and the range assignor moves 58 partitions in all: 720 partition-seconds of no consumption and a peak lag of 6,000 messages. Cooperative sticky rebalancing pauses only the 24 partitions that actually move: 120 partition-seconds, peak lag 1,000. Static membership avoids rebalancing at all, since each consumer is back within the session timeout; only its own partitions wait for it: 240 partition-seconds and a peak lag of 4,000, a little worse than cooperative when nothing has state to restore.
What each strategy does
- Eager rebalancing stops all 12 partitions twice per restart, once when the consumer leaves and once when it returns, and the range assignor moves 58 partitions over the deploy, many of them needlessly. Stateless, that is 720 partition-seconds of no consumption; with 60 s state restores, a peak lag of 179,500 messages that takes until 17.6 minutes after the deploy started to clear.
- Cooperative sticky rebalancing pauses only the partitions that move, 24 in all: 120 partition-seconds stateless. With state, each moved partition still waits for its restore, so the peak lag is 66,500.
- Static membership treats a consumer that comes back within the session timeout as the same member: no rebalance and no state to move. Its own two partitions wait while it restarts, 240 partition-seconds, a little more than cooperative when consumers are stateless and far less when they are stateful.
The settings
# cooperative incremental rebalancing: only moved partitions are revoked
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor
# static membership: a restart within the session timeout is not a membership change
group.instance.id=orders-consumer-3 # stable per instance, e.g. the pod name
session.timeout.ms=45000 # longer than a restart takes
# Kafka Streams: keep warm copies of state elsewhere so a moved task starts quickly
num.standby.replicas=1Rebalances are also triggered by consumers that stop polling, a slow batch that exceeds max.poll.interval.ms looks like a dead member, so keeping processing per poll short avoids rebalances that look random.
In a system design interview
When a design has a large consumer group, especially a stateful stream processor, say how deploys avoid stalls: “Consumers use static membership with a session timeout longer than a restart, so rolling deploys do not trigger rebalances, and the cooperative sticky assignor for real scaling events, so only the moved partitions pause. Stateful tasks keep a standby replica of their state so a moved partition resumes in seconds instead of replaying its changelog.” Expect the follow-ups: what happens to offsets during a rebalance, how to scale the group beyond the partition count, and how lag is monitored.
Read the guide to Kafka internals or deployment strategies, see Kafka partitions and consumer groups and graceful shutdown, or watch a strong candidate design a message queue.
Questions people ask
What is a Kafka consumer group rebalance?
The process by which the members of a consumer group agree on which member consumes which partitions. It runs whenever a member joins, leaves or is considered dead, and when partitions are added to a subscribed topic.
Why are eager rebalances disruptive?
With the eager protocol every member revokes all of its partitions at the start of a rebalance and consumes nothing until the new assignment arrives. Every membership change, including each step of a rolling deploy, briefly stops the whole group.
What is cooperative incremental rebalancing?
A protocol, used by the CooperativeStickyAssignor, in which members keep consuming the partitions they will still own and give up only those that must move, possibly over two rebalance rounds. Most partitions never pause.
What is static membership in Kafka?
Giving each consumer a stable group.instance.id. When a static member restarts and rejoins within session.timeout.ms, the coordinator treats it as the same member, so no rebalance happens and it gets the same partitions back.
Why do rebalances hurt stateful stream processing more?
A partition that moves to another instance needs its state, such as a Kafka Streams state store, which may have to be rebuilt from a changelog topic before processing resumes. Standby replicas and sticky assignment keep that rare and short.
What causes unexpected rebalances?
Consumers that miss heartbeats because of long garbage collection pauses or network trouble, and consumers that take longer than max.poll.interval.ms between polls because a batch of processing was slow, both look dead to the coordinator.
How do you monitor rebalances?
Track consumer lag per partition, the rebalance rate and duration from client metrics, and assignment changes in logs. Lag that spikes on every deploy is the usual sign that rebalances are stopping consumption.