Updated 24 October 2026

Distributed transactions and idempotency

Why cross-service transactions are hard, two-phase commit vs sagas, the outbox pattern, idempotency keys, exactly-once semantics, and how to keep money and inventory correct.

Try it yourself, in a playground:Two-phase commitIdempotency keysOptimistic vs pessimistic lockingTransactional outboxSaga patternDouble-entry ledger

Inside one relational database, a transaction makes several changes atomic, and it is easy. Across two services, two databases, or a database and a message broker, there is no transaction. Every design that moves money, reserves inventory, or updates two systems together must answer how it stays correct when one write succeeds and the other fails. This is the deep dive in payments, ordering, booking and messaging questions.

The problem

An order placement writes the order, decrements inventory, charges the card and sends a confirmation. If these are four services, any step can fail after the previous ones succeeded, and the caller can crash or time out between any two. Without a strategy you end up with charged customers without orders, orders without reserved stock, or duplicate charges from retries.

Two-phase commit

A coordinator asks every participant to prepare (do the work, hold locks, promise to commit), and if all say yes, tells them to commit; otherwise abort. It gives atomicity across resources, and it is what XA transactions and some distributed databases use internally. The costs are why it is rarely used between services: participants hold locks while waiting for the coordinator, which becomes a latency and availability bottleneck; if the coordinator dies after prepare, participants are blocked until it returns; and it requires every participant to support the protocol, which third parties (a payment provider) do not. Mention it, say why not, move on.

Sagas

A saga is a sequence of local transactions, each in one service, where every step has a compensating action that undoes it. Reserve inventory → charge card → confirm order. If the charge fails, release the inventory. If the confirmation fails after the charge, refund. There is no atomicity: other requests can observe intermediate states (stock reserved but order not confirmed), so states must be explicit (pending, confirmed, cancelled) and reads must treat them correctly.

Two ways to run one:

• Choreography. Each service listens for the previous step's event and emits its own. No central coordinator, but the flow is implicit and hard to follow past four or five steps.
• Orchestration. A saga orchestrator (a workflow engine like Temporal, or a service with a state table) calls each step and records progress durably, so a crash resumes from the last completed step. Explicit, observable, and the choice for anything involving money.

Compensation is not always possible (an email cannot be unsent), so order the steps with the least reversible last: reserve, charge, then notify.

The outbox pattern

The most common dual-write bug: commit to the database, then publish an event, and crash in between. Write the event into an outbox table in the same transaction as the business row; a relay publishes rows from the outbox to the broker and marks them sent (at-least-once). Now the database and the stream cannot disagree. Combine with idempotent consumers and you have a reliable, exactly-once-effective pipeline without distributed transactions. See queues and streams.

Idempotency

An operation is idempotent if doing it twice has the same effect as once. Retries, redeliveries and failover all cause duplicates, so every write that matters must be idempotent.

Idempotency keys. The client generates a unique key for each logical operation (a UUID per checkout attempt) and sends it with every retry. The server atomically records the key before doing the work (INSERT … ON CONFLICT DO NOTHING, or Redis SET NX), and if it already exists, returns the stored result. Stripe stores keys for 24 hours; a shorter window (a few hours) covers realistic retries. The key must be scoped to the caller (per API key) so two customers cannot collide, and the stored response should be returned as-is, including errors, so the client sees consistent behaviour.

Natural keys. Many operations have one already: "mark order 123 shipped" is idempotent because it sets a state; "send confirmation for order 123" is idempotent if the sender records order id. Prefer operations shaped as "set to X" over "add N".

Versioning / optimistic concurrency. Include the version the client read (If-Match: etag or a version column); the write succeeds only if unchanged. Prevents lost updates from concurrent writers, which is a different problem from retries but often asked together.

Exactly-once

End-to-end exactly-once across arbitrary systems does not exist; what exists is at-least-once delivery plus idempotent processing, which produces exactly-once effects. Kafka's transactional producer gives exactly-once for Kafka-to-Kafka processing by making the offset commit and the output write atomic. Stream processors (Flink, Kafka Streams) build on that. When the interviewer says "exactly once", say this and explain where the dedupe lives.

Keeping money correct

Money designs add a few rules that interviewers expect to hear:

• Double-entry ledger. Every movement is two entries (debit one account, credit another) that sum to zero, appended and never updated. Balances are derived (sum of entries, cached) not stored as the source of truth, so a bug cannot silently create money.
• State machine with an unknown state. A charge is created → pending → succeeded | failed, and a provider timeout leaves it unknown, which must be resolved by querying the provider, never by assuming failure and retrying without the same idempotency key.
• Integer minor units, never floats. Amounts are 4999 cents.
• Reconciliation. A daily job compares your ledger against the provider's report and flags every difference. Say this; it is how real systems catch what idempotency and sagas miss.

Inventory and reservations

Reserve with a TTL rather than decrementing on add-to-cart: reserved_until on the stock row or a Redis key with expiry, released by the saga on failure or by expiry on abandonment. Decrement the real count on payment success. Oversell is prevented by an atomic conditional update (UPDATE stock SET available = available - 1 WHERE id = ? AND available > 0) rather than read-then-write. For flash sales with one hot row, move the counter to Redis (DECR, check ≥ 0) and reconcile to the database asynchronously.

In the interview

"Order placement is an orchestrated saga: reserve inventory with a 10-minute TTL, charge via the payment service with an idempotency key from the client, confirm the order. Each step writes its state change and an outbox event in one transaction; a relay publishes to Kafka. Failure after charge triggers a compensating refund. Consumers dedupe on event id. A daily reconciliation job compares the ledger with the provider's settlement file." Every follow-up (crash between steps, duplicate request, provider timeout) is answered by one of those sentences.

Choosing between them

A quick decision path you can say out loud:

1. Can the whole operation live in one database? Then use a local transaction, and stop. Most "distributed transaction" questions are really "should these two services share a database" questions, and sometimes the answer is that they should be one service.
2. Does every step have a compensating action a business person would accept? (Refund, release, cancel, un-reserve.) Then a saga.
3. Is the operation short, inside one trust boundary, and correctness-critical with no sensible compensation? Then 2PC, and accept the coupling.
4. Is the requirement only "do this eventually and exactly once in effect"? Then an outbox plus idempotent consumers, which is the most common real answer.

Testing the hard cases

Interviewers ask "how do you know this works". The three tests worth naming:

• Duplicate delivery: run the same message twice and assert the end state is identical, including any counters and any outbound side effects.
• Crash between steps: kill the worker after the local commit and before the publish, and verify the outbox relay still produces the event exactly once in effect.
• Out-of-order arrival: deliver step 3's event before step 2's, and check the state machine rejects the transition rather than corrupting state.

Checklist

• The steps, and which of them are local transactions.
• The idempotency key, where it is stored, and its retention.
• The outbox (or equivalent) that makes the event durable with the write.
• The compensation for every step that can fail after an earlier one committed.
• What the user sees while the operation is in flight, and what they see if it ultimately fails.

Questions people ask

What is idempotency in distributed systems?

An operation is idempotent if doing it twice has the same effect as doing it once. In distributed systems, retries, redeliveries and failover all cause duplicates, so every write that matters must be idempotent. Prefer operations shaped as "set to X" over "add N": "mark order 123 shipped" is idempotent because it sets a state, and "send confirmation for order 123" is idempotent if the sender records the order id.

What is an idempotency key?

An idempotency key is a unique key the client generates for each logical operation, such as a UUID per checkout attempt, and sends with every retry. The server atomically records the key before doing the work (INSERT … ON CONFLICT DO NOTHING, or Redis SET NX) and, if the key already exists, returns the stored result, including errors. Stripe stores keys for 24 hours, though a few hours covers realistic retries, and keys are scoped to the caller so two customers cannot collide.

How do you get exactly-once from at-least-once delivery?

Add idempotent processing to at-least-once delivery: a message may arrive more than once, but it takes effect only once. End-to-end exactly-once across arbitrary systems does not exist, so what you build is exactly-once effects, with the dedupe in the consumer, for example on event id. Within Kafka, the transactional producer gives exactly-once for Kafka-to-Kafka processing by making the offset commit and the output write atomic.

Two-phase commit vs saga: what is the difference?

Two-phase commit makes a change atomic across resources, while a saga gives up atomicity and undoes failed steps with compensating actions. In 2PC, participants prepare and hold locks until a coordinator says commit or abort, so the coordinator becomes a latency and availability bottleneck, and third parties such as a payment provider do not support the protocol. Use a saga when every step has a compensation the business would accept, and 2PC only for short, correctness-critical operations inside one trust boundary with no sensible compensation.

What is a saga in distributed transactions?

A saga is a sequence of local transactions, each in one service, where every step has a compensating action that undoes it. An order might reserve inventory, charge the card, then confirm: if the charge fails, the inventory is released, and if confirmation fails after the charge, the customer is refunded. Other requests can see intermediate states, so states must be explicit (pending, confirmed, cancelled), and the least reversible step runs last, because an email cannot be unsent.

What is the outbox pattern?

The outbox pattern writes an event into an outbox table in the same transaction as the business row, then a relay publishes those rows to the broker and marks them sent. It fixes the most common dual-write bug: committing to the database, then crashing before the event is published. The relay is at-least-once, so combined with idempotent consumers it gives a reliable, exactly-once-effective pipeline in which the database and the stream cannot disagree, without distributed transactions.