Kafka partitions and consumer groups: an interactive lag simulator

A Kafka topic is split into partitions, and a consumer group shares them out so that each partition is read by exactly one consumer in the group. Partitions are the unit of parallelism: a group can never use more consumers than there are partitions, a key always goes to the same partition so its messages stay in order, and one overloaded partition falls behind on its own while the rest keep up.

The playground sends 480 messages a second into a topic and reads them with a group of consumers that can each process 120 a second. Change the partitions and consumers, add a hot key or crash a consumer, and watch each partition’s lag over a minute.

How partitions and consumer groups work

1. Produce: a message with a key goes to partition hash(key) mod partitions; one without a key is spread across partitions.
2. Store: each partition is an ordered, append-only log, and each message gets the next offset in it. Order is guaranteed within a partition, not across them.
3. Assign: consumers with the same group id split the partitions between them. Each partition has one owner in the group; another group reads the same messages independently.
4. Consume and commit: a consumer reads its partitions in order and commits the offset it has processed up to, so a replacement knows where to resume.
5. Lag: the newest offset minus the committed offset, the number of messages written but not yet processed.

Choosing the number of partitions

The partition count caps the group’s parallelism. In the third preset, eight consumers read six partitions and two of them sit idle. It also decides how evenly work divides: with six partitions and four consumers, two consumers get two partitions each, 160 messages a second against the 120 they can handle, and their partitions end the minute about 1,200 messages behind while the other two consumers have room to spare.

So choose a count with headroom for the most consumers you expect, and one that divides evenly by the common group sizes: 12 splits between 1, 2, 3, 4, 6 or 12 consumers. Adding partitions later is possible but changes hash(key) mod partitions, so existing keys move to new partitions and their order across the change is lost. More partitions also cost more open files, more replication traffic and longer leader elections, so thousands per topic is rarely wise.

Hot partitions

Keying by user keeps each user’s events in order, but a single busy key lands entirely on one partition. In the hot key preset one user sends 30% of all messages, so P1 receives 200 a second, more than any one consumer can process, and its lag reaches about 4,800 by the end of the minute. Adding consumers changes nothing, because a partition is never shared within a group. The fixes trade ordering or effort: split the hot key by adding a suffix, which spreads it over several partitions but loses its ordering across them; route known heavy hitters to their own topic; or make each message cheaper to process, for example by batching.

Rebalances

When a consumer joins, leaves or stops sending heartbeats, the group rebalances and reassigns partitions. In the crash preset, a consumer dies at 20 seconds; its partition is unread until the group notices 6 seconds later, then every consumer stops for a 2 second rebalance, and afterwards the survivors take over its share, which leaves one of them with two partitions to read. Real detection takes as long as the session timeout, 45 seconds by default in recent Kafka versions.

The pause is the part to design around. Cooperative rebalancing, with the CooperativeStickyAssignor, moves only the partitions that need a new owner and lets the rest keep working. Static membership, setting a group.instance.id, lets a consumer restart within the session timeout without any rebalance, which matters for rolling deploys.

Consumer lag is the number to watch

Lag tells you whether consumers keep up, and lag divided by the arrival rate tells you how far behind they are in time. Alert when it grows steadily rather than when it spikes, since a rebalance or a deploy causes a brief rise that drains on its own. Autoscalers such as KEDA can add consumers as lag grows, up to the partition count.

Consumer group settings in practice

# producer: the key decides the partition, so one user's events stay in order
producer.send("events", key=user_id, value=event)

# consumer: every process started with the same group.id shares the partitions
group.id = "billing"
partition.assignment.strategy = CooperativeStickyAssignor   # move only what must move
group.instance.id = "billing-7"         # static membership: a restart is not a rebalance
session.timeout.ms = 45000              # how long before a silent consumer is declared dead
enable.auto.commit = false              # commit offsets after processing, not before

In a system design interview

State the key, the count and the reason: “Events are keyed by account id, so each account’s events are processed in order. We create 48 partitions: at 20,000 events a second and 1,000 a second per consumer we need 20 consumers today, and 48 leaves room to more than double. We alert on consumer lag in seconds.” Expect the follow-ups: a hot account, what happens when a consumer dies mid-batch (offsets are committed after processing, so the batch is processed again and the handler must be idempotent), and how to add partitions without breaking ordering.

Read the guide to Kafka internals, try queueing and Little’s law to size the consumers, or watch a strong candidate design an ad click aggregator, where partitioning the click stream is a central choice.

Questions people ask