Distributed transactions and idempotency

Why cross-service transactions are hard, two-phase commit vs sagas, the outbox pattern, idempotency keys, exactly-once semantics, and how to keep money and inventory correct.

Inside one relational database, a transaction makes several changes atomic, and it is easy. Across two services, two databases, or a database and a message broker, there is no transaction. Every design that moves money, reserves inventory, or updates two systems together must answer how it stays correct when one write succeeds and the other fails. This is the deep dive in payments, ordering, booking and messaging questions.

The problem

An order placement writes the order, decrements inventory, charges the card and sends a confirmation. If these are four services, any step can fail after the previous ones succeeded, and the caller can crash or time out between any two. Without a strategy you end up with charged customers without orders, orders without reserved stock, or duplicate charges from retries.

Two-phase commit

A coordinator asks every participant to prepare (do the work, hold locks, promise to commit), and if all say yes, tells them to commit; otherwise abort. It gives atomicity across resources, and it is what XA transactions and some distributed databases use internally. The costs are why it is rarely used between services: participants hold locks while waiting for the coordinator, which becomes a latency and availability bottleneck; if the coordinator dies after prepare, participants are blocked until it returns; and it requires every participant to support the protocol, which third parties (a payment provider) do not. Mention it, say why not, move on.

Sagas

A saga is a sequence of local transactions, each in one service, where every step has a compensating action that undoes it. Reserve inventory → charge card → confirm order. If the charge fails, release the inventory. If the confirmation fails after the charge, refund. There is no atomicity: other requests can observe intermediate states (stock reserved but order not confirmed), so states must be explicit (pending, confirmed, cancelled) and reads must treat them correctly.

Two ways to run one:

• Choreography. Each service listens for the previous step's event and emits its own. No central coordinator, but the flow is implicit and hard to follow past four or five steps.
• Orchestration. A saga orchestrator (a workflow engine like Temporal, or a service with a state table) calls each step and records progress durably, so a crash resumes from the last completed step. Explicit, observable, and the choice for anything involving money.

Compensation is not always possible (an email cannot be unsent), so order the steps with the least reversible last: reserve, charge, then notify.

The outbox pattern

The most common dual-write bug: commit to the database, then publish an event, and crash in between. Write the event into an outbox table in the same transaction as the business row; a relay publishes rows from the outbox to the broker and marks them sent (at-least-once). Now the database and the stream cannot disagree. Combine with idempotent consumers and you have a reliable, exactly-once-effective pipeline without distributed transactions. See queues and streams.

Idempotency

An operation is idempotent if doing it twice has the same effect as once. Retries, redeliveries and failover all cause duplicates, so every write that matters must be idempotent.

Idempotency keys. The client generates a unique key for each logical operation (a UUID per checkout attempt) and sends it with every retry. The server atomically records the key before doing the work (INSERT … ON CONFLICT DO NOTHING, or Redis SET NX), and if it already exists, returns the stored result. Stripe stores keys for 24 hours; a shorter window (a few hours) covers realistic retries. The key must be scoped to the caller (per API key) so two customers cannot collide, and the stored response should be returned as-is, including errors, so the client sees consistent behaviour.

Natural keys. Many operations have one already: "mark order 123 shipped" is idempotent because it sets a state; "send confirmation for order 123" is idempotent if the sender records order id. Prefer operations shaped as "set to X" over "add N".

Versioning / optimistic concurrency. Include the version the client read (If-Match: etag or a version column); the write succeeds only if unchanged. Prevents lost updates from concurrent writers, which is a different problem from retries but often asked together.

Exactly-once

End-to-end exactly-once across arbitrary systems does not exist; what exists is at-least-once delivery plus idempotent processing, which produces exactly-once effects. Kafka's transactional producer gives exactly-once for Kafka-to-Kafka processing by making the offset commit and the output write atomic. Stream processors (Flink, Kafka Streams) build on that. When the interviewer says "exactly once", say this and explain where the dedupe lives.

Keeping money correct

Money designs add a few rules that interviewers expect to hear:

• Double-entry ledger. Every movement is two entries (debit one account, credit another) that sum to zero, appended and never updated. Balances are derived (sum of entries, cached) not stored as the source of truth, so a bug cannot silently create money.
• State machine with an unknown state. A charge is created → pending → succeeded | failed, and a provider timeout leaves it unknown, which must be resolved by querying the provider, never by assuming failure and retrying without the same idempotency key.
• Integer minor units, never floats. Amounts are 4999 cents.
• Reconciliation. A daily job compares your ledger against the provider's report and flags every difference. Say this; it is how real systems catch what idempotency and sagas miss.

Inventory and reservations

Reserve with a TTL rather than decrementing on add-to-cart: reserved_until on the stock row or a Redis key with expiry, released by the saga on failure or by expiry on abandonment. Decrement the real count on payment success. Oversell is prevented by an atomic conditional update (UPDATE stock SET available = available - 1 WHERE id = ? AND available > 0) rather than read-then-write. For flash sales with one hot row, move the counter to Redis (DECR, check ≥ 0) and reconcile to the database asynchronously.

In the interview

"Order placement is an orchestrated saga: reserve inventory with a 10-minute TTL, charge via the payment service with an idempotency key from the client, confirm the order. Each step writes its state change and an outbox event in one transaction; a relay publishes to Kafka. Failure after charge triggers a compensating refund. Consumers dedupe on event id. A daily reconciliation job compares the ledger with the provider's settlement file." Every follow-up (crash between steps, duplicate request, provider timeout) is answered by one of those sentences.