Message ordering: an interactive simulator of parallel consumers, partitions and retries

Events are produced in order, but nothing guarantees they are processed in order. Parallel consumers, retries and multiple partitions all let a later event overtake an earlier one, so an order can go from shipped back to paid. Systems that need order get it per key: route each entity’s events through one partition, or make the consumer reject events older than what it has already applied.

The playground sends five orders’ status events through four consumers three ways. Turn on a failed event that is retried, or shuffle the processing times, and compare the final statuses.

Three ways order breaks

• Parallel consumers on one queue. Two consumers take an order’s paid and shipped events a moment apart; if paid takes longer, shipped lands first and paid then overwrites it. In the playground, a shared queue sends 2 events in after newer ones, and one order ends at shipped instead of delivered.
• Retries. A failed event sent to the back of the queue, or to a retry topic, is processed after its successors. Even partitioning by order id does not help: the retried paid event lands after shipped and delivered, and the order ends at paid.
• Producers. A producer retrying sends with several requests in flight can reorder them; Kafka’s idempotent producer keeps them in order.

Getting order back

• Partition by key. Kafka keeps order within a partition, SQS FIFO within a message group, and Pub/Sub within an ordering key. Use the entity id as the key and one consumer handles each entity at a time. The price is parallelism: in the playground the partitioned run finishes at 18 instead of 13, because a busy consumer cannot hand its orders to an idle one, and one hot key can hold up its whole partition.
• Retry in place. Keep retrying the failed event before moving on, or park every later event for that key until it succeeds, rather than skipping ahead.
• Version the writes. Give every event a sequence number per entity and apply it only if it is newer than the stored version. Order no longer matters for the final state: in the playground every order ends delivered, at the shared queue’s speed, with stale events dropped.
• Prefer commutative updates where you can: increments and set unions give the same result in any order.

Global order across all events is rarely needed and does not scale, since it allows only one consumer. Ask which events must be ordered relative to each other; usually it is only events about the same thing.

Keyed producing and versioned writes

# producer: key by order id, so all of an order's events go to one partition, in order
producer = Producer({"bootstrap.servers": BROKERS, "enable.idempotence": True})
producer.produce("order-events", key=str(order_id), value=json.dumps(event))

-- consumer: apply an event only if it is newer than what the row reflects
UPDATE orders
SET status = :status, version = :seq
WHERE id = :order_id AND version < :seq;
-- 0 rows updated: a newer event already landed, so this one is dropped

In a system design interview

State the ordering you need and how you get it: “Order events are keyed by order id, so each order’s events stay in one partition and one consumer applies them in sequence. Each event carries a per-order version, and the status update is conditional on it, so a replay or a late retry cannot move an order backwards. Failures for an order block only that order, in a retry queue keyed the same way.” Expect the follow-ups: hot keys, what happens when partitions are added and keys move, and ordering across two different entities.

Read the guide to message ordering or Kafka internals, see Kafka partitions and consumer groups and delivery semantics, or watch a strong candidate design a message queue.

Questions people ask