At-most-once, at-least-once and exactly-once: an interactive delivery simulator

A message system can promise to deliver each message at most once, which may lose some, or at least once, which may repeat some. Exactly once delivery across a network is not possible in general, because a consumer can always crash between doing the work and recording that it did. What systems offer instead is effectively once processing: at-least-once delivery, plus consumers that make repeats harmless.

The playground runs 200 messages, each crediting an account, through a consumer that crashes now and then and restarts from its last committed position. Each grid shows what happened to every message.

Where the guarantee comes from

A consumer does two things for every message: the work, and recording its position in the stream, a Kafka offset commit or an SQS delete. A crash can fall between them, and the order decides the outcome:

• At most once: record the position first, then do the work. A crash in between skips the message forever. With commits every 10 messages and 2% crashes, the playground loses 10 of 200 credits.
• At least once: do the work first, then record the position. A crash in between redelivers the batch on restart, and the work runs again: 20 credits applied twice in the same run.
• Effectively once: at-least-once delivery, with the work made idempotent. Every one of the 200 credits is applied exactly once, even though the consumer handled 222 deliveries.

Making repeats harmless

• Record processed message ids in the same database transaction as the effect, and skip any id already recorded. This is what the playground does.
• Make the operation naturally idempotent: “set status to shipped” or an upsert by a unique key can safely run twice; “add $40” cannot.
• Use versions: apply an update only if the record’s version is older than the event’s.
• Store the offset with the result: if the output and the consumer’s position commit in one transaction, a crash rolls back both.

Kafka’s exactly-once support does the last of these within Kafka: an idempotent producer drops duplicate sends, and transactions commit the output messages and the consumer’s offsets atomically. It covers a read-process-write pipeline from Kafka to Kafka, not side effects in other systems, such as charging a card or sending an email, which still need idempotency of their own.

Batch size

Committing after every message keeps the damage of a crash to one message but costs a round trip each time; committing every 50 is cheaper but replays or loses up to 50. In the playground, at-least-once with commits every 50 applies 77 messages more than once. Most consumers commit every few hundred milliseconds or few hundred messages and rely on idempotency for the rest.

Effectively-once processing in code

# effectively once: the side effect and "I have handled this message" commit together
def handle(msg, db):
    with db.transaction() as tx:
        inserted = tx.execute(
            "INSERT INTO processed (message_id) VALUES (%s) ON CONFLICT DO NOTHING",
            (msg.id,),
        ).rowcount
        if inserted == 0:
            return                                    # seen before: a redelivery, skip it
        tx.execute("UPDATE accounts SET balance = balance + %s WHERE id = %s",
                   (msg.amount, msg.account_id))

for batch in consumer.poll(max_records=10):
    for msg in batch:
        handle(msg, db)
    consumer.commit()                                 # after processing: at least once
# a crash before commit replays the batch; handle() makes the replay harmless

In a system design interview

Never claim exactly-once delivery; say how duplicates are handled: “Payment events are consumed at least once: the consumer commits offsets after processing. Each event carries an id, and the ledger insert uses it as a unique key in the same transaction as the balance update, so a redelivered event is a no-op. Losing an event is worse than seeing it twice, so we never commit before processing.” Expect the follow-ups: how long processed ids are kept, ordering within a key, and what happens with side effects outside your database.

Read the guide to delivery semantics, see idempotency keys for the same idea in APIs and the transactional outbox for the producing side, or watch a strong candidate design a message queue.

Questions people ask