Dead-letter queues and poison messages: an interactive simulator

Some messages can never be processed: malformed payloads, references to deleted records, a bug triggered by one customer’s data. These poison messages fail on every attempt. Retrying them forever blocks or slows everything else, so queues move a message to a dead-letter queue (DLQ) after a few failed attempts, where it waits for a person or a tool to inspect it, fix the cause and replay it.

The playground runs a consumer for ten minutes against a queue where a small share of messages are poison. Compare retrying in place, requeueing forever, and a dead-letter queue.

Two ways poison hurts

• Head-of-line blocking. A consumer that must process messages in order, such as a Kafka partition consumer, keeps retrying the message at the head. With 2% poison, the first one arrives within seconds and nothing behind it is processed again: 14 messages handled in ten minutes, 3,500 waiting.
• Endless recirculation. Requeueing failures at the back keeps good messages flowing, but poison never leaves. By the end, 78 poison messages are circling, each costing a timeout on every lap, and good messages wait over half a minute. Given longer, they would use up the consumer entirely.

A dead-letter queue bounds the damage: after 3 attempts a message is set aside, the consumer moves on, and good messages wait well under a second. The 78 poison messages are not lost; they sit in the DLQ with their error, for someone to fix and replay.

Running a dead-letter queue well

• Separate transient from permanent failures. A timeout talking to a database deserves retries with backoff, ideally on a delay queue so they do not block others; a schema violation should go to the DLQ at once.
• Alert on DLQ depth. A DLQ nobody watches is a slow data-loss bug. Page or ticket when messages land, and keep the error and attempt count with each one.
• Make replay safe. Replayed messages arrive late and out of order, so handlers must be idempotent and tolerate stale data.
• Mind ordering. Skipping a message breaks order for its key; if order matters, park the whole key until the poison message is resolved.

A dead-letter queue in SQS and Kafka

# SQS: after 3 failed receives, the message moves to the dead-letter queue
aws sqs set-queue-attributes --queue-url $ORDERS_QUEUE --attributes '{
  "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:eu-west-1:123456789012:orders-dlq\",\"maxReceiveCount\":\"3\"}"
}'

# Kafka has no built-in DLQ; consumers publish failures to their own topic
def consume(msg):
    try:
        handle(msg)
    except PermanentError:                   # bad data: retrying will not help
        dlq.send("orders.dlq", msg.value, headers={"error": traceback.format_exc()})
    except TransientError:
        retry_topic.send("orders.retry-1m", msg.value)   # retry later, off the main path
    consumer.commit(msg)                     # either way, the partition moves on

In a system design interview

Whenever you put a queue in a design, say what happens to a message that keeps failing: “Consumers retry transient errors three times with backoff through a delay topic. Anything still failing, and anything that fails validation, goes to a dead-letter topic with the error attached. DLQ depth pages the owning team, and a replay tool re-publishes fixed messages to the main topic; handlers are idempotent, so replays are safe.” Expect the follow-ups: ordering for the affected key, how long DLQ messages are kept, and alerting.

Read the guide to message queues and streams or background jobs, see message ordering and delivery semantics, or watch a strong candidate design a message queue.

Questions people ask