Message queues and event streams

When to go asynchronous, queues vs logs (SQS, RabbitMQ vs Kafka), delivery guarantees, ordering, consumer groups, dead-letter queues, backpressure, and the outbox pattern.

A queue lets one part of the system hand work to another without waiting for it. That single property buys you decoupling (the producer does not know or care who consumes), buffering (a burst is absorbed and drained at the consumer's pace), and resilience (a consumer outage delays work instead of failing requests). Nearly every design has an asynchronous path, and interviewers ask three things about it: what goes through it, what happens when a message is delivered twice, and what happens when consumers fall behind.

What to make asynchronous

Anything the user does not need to see in the response: sending email and push notifications, resizing images, updating search indexes and caches, fanning out a post to followers, writing analytics, calling slow third parties, and any retryable side effect. The rule: if the request can return success before this work is done, and the work can be retried safely, it goes on a queue.

Keep synchronous: the write to the system of record, authorisation, anything the user must see immediately (their own message in the chat), and anything whose failure must fail the request (payment authorisation).

Queues versus logs

Queues (SQS, RabbitMQ, Google Pub/Sub in its default mode) deliver each message to one consumer and delete it when acknowledged. They model tasks: each message is a job to be done once. Good for work distribution, retries with visibility timeouts, and priority. No replay: once consumed, it is gone.

Logs (Kafka, Pulsar, Kinesis, Redpanda) append messages to a partitioned, ordered, durable log that consumers read at their own offset. Messages are retained for a period (days to forever) regardless of consumption. Many independent consumer groups can read the same stream; a consumer can rewind and replay. They model events: things that happened, which several systems want to know about. Higher throughput (millions of messages per second per cluster) and ordering per partition.

Choose a queue for jobs with one consumer; choose a log when multiple consumers need the same events, when replay matters (rebuilding a cache or index), or when order within a key matters. In interviews, "Kafka" is the default for event streams and "SQS" for job queues, and it is fine to say both are in the design.

Delivery guarantees

• At most once. Fire and forget. Messages can be lost. Acceptable only for metrics and similar.
• At least once. The broker redelivers until acknowledged; consumers must handle duplicates. This is what every practical system provides.
• Exactly once. Not achievable end to end across arbitrary systems. Kafka offers exactly-once within a Kafka-to-Kafka pipeline via transactions, and some sinks support it via idempotent writes. In practice "exactly once" means at-least-once delivery plus idempotent processing.

So the real question is how the consumer is idempotent: a unique message id checked against a processed set (a table or Redis set with TTL), a natural idempotency key (the order id for "send confirmation"), or an idempotent operation (set status = shipped, not increment count). See idempotency.

Ordering

A queue does not guarantee order in general; a log guarantees order within a partition. Choose the partition key so that everything that must be ordered shares a key: user id, channel id, order id. Then the events for one order arrive in order, while different orders are processed in parallel. Ordering across keys is not needed and not worth the cost. If the interviewer asks about global ordering, the answer is "one partition, and it will not scale; instead I make consumers tolerant of reordering across keys".

Consumers and scaling

A consumer group shares a topic: each partition is read by exactly one consumer in the group. Parallelism is therefore bounded by partition count; choose it generously up front (partitions are cheap, resharding is not) and aim for 10 to 100 per topic. Each consumer commits its offset after processing; a crash means the next consumer replays from the last commit, which is where duplicates come from.

Backpressure is what happens when producers outrun consumers. With a log, the lag grows (offset distance) and retention protects you for days; monitor consumer lag and autoscale consumers up to the partition count. With a queue, the queue depth grows; set alarms on depth and age of oldest message. If consumers cannot catch up, you either add partitions and consumers, batch the work, or shed load by dropping low-priority messages.

Failure handling

• Retries. A failed message is retried with exponential backoff. Distinguish transient failures (retry) from permanent ones (bad data: do not retry forever).
• Dead-letter queue. After N failures, move the message aside for inspection so one poison message does not block the partition. Alert on DLQ depth.
• Visibility timeout. With SQS-style queues, a message being processed is hidden for a timeout; if the consumer crashes, it reappears. Set the timeout longer than processing time or you get duplicates under normal operation.
• Idempotent consumers are the answer to all of the above.

The outbox pattern

The subtle bug: a service writes to its database and then publishes an event, and crashes in between. The database has the order; the stream does not. Or the publish succeeds and the commit fails. Dual writes to two systems cannot be atomic.

The fix is the transactional outbox: write the event to an outbox table in the same database transaction as the business change. A separate relay reads the outbox (polling or via CDC) and publishes to the stream, marking rows sent. Publishing is at-least-once, so consumers are idempotent. This pattern comes up in every payments, ordering and messaging design; say it unprompted.

The alternative is change data capture: skip the outbox and publish the database's own change log (Debezium). Less code, but the event schema is your table schema.

Sizing

Kafka: a partition handles roughly 10 MB/s and tens of thousands of messages per second; a broker handles thousands of partitions; a modest 6-broker cluster does a million messages per second. Retention is a disk calculation: 100 k msgs/s × 1 KB × 7 days ≈ 60 TB with replication factor 3 (before compression). SQS is effectively unlimited with a per-queue throughput in the thousands per second and no retention beyond 14 days.

In the interview

"After committing the message to the store, the service writes an event to an outbox table in the same transaction; a relay publishes it to Kafka keyed by channel id, so a channel's events stay ordered. Fan-out workers in a consumer group read it and push to connected clients; a second consumer group updates the search index. Consumers are idempotent on message id. Lag is monitored and consumers scale to the partition count; poison messages go to a DLQ after three retries."