System design interview question

Design a Payment System

Q: What exactly is stored under an idempotency key, for how long, and what happens on every kind of conflict?

Client-generated UUID per attempt, scoped by (merchant_id, key), stored durably with the request fingerprint, a state (in_progress → done) and the final response, TTL 24 hours (long enough for any sane retry, short enough to bound storage). Reserve atomically before any side effect. Conflict rules: same key + same fingerprint + done → return cached response; same key + in_progress → 409 with retry-after; same key + different fingerprint → 422, never execute. Stale in_progress keys older than the PSP timeout are handed to a recovery worker that queries the PSP for the outcome. Forward the same key to the PSP as belt and braces.

Q: Why not just keep a balance column per merchant and add or subtract from it?

Append-only journal in Postgres: journal_entries (id, event_id unique, created_at, description) and postings (entry_id, account_id, amount signed, currency). A deferred constraint checks that postings per entry sum to zero. The application role has INSERT and SELECT only. Balances come from a nightly snapshot table plus a sum of postings since the snapshot, which keeps reads fast without giving up derivation. Hot accounts (the platform fee account receives a posting on every payment) are the scaling concern: because postings are append-only there is no row contention, only index growth, and snapshots keep reads bounded. Idempotency on event_id means a replayed event cannot post twice.

Q: The PSP call timed out. Did the charge happen? What do we do now?

States: created → authorizing → authorized → capturing → captured → (refunding → refunded) with failed reachable only from a definitive PSP failure and a terminal "requires_review" for contradictions. A timeout leaves the payment in authorizing. A recovery worker picks up payments in authorizing older than the PSP timeout and calls get-by-idempotency-key at the PSP: found and succeeded → authorized; found and failed → failed; not found → safe to retry the authorize with the same key, or fail after a bounded number of attempts. Every transition takes a row lock and validates the from-state, so a late webhook and the recovery worker cannot both apply. Time-in-authorizing is an alerting metric.

Q: The payment row committed but the Kafka publish failed. Now the ledger and the merchant never hear about it. How do we make "state changed" and "event published" atomic?

Every state transition inserts into an outbox table (event_id, aggregate_id, type, payload, created_at) in the same transaction as the payment row. A relay publishes unpublished rows to Kafka keyed by payment_id and marks them sent; CDC from the Postgres WAL is the higher-throughput version of the same idea. Consumers (ledger, webhooks, analytics) each track processed event ids so redelivery is harmless. This gives at-least-once delivery with exactly-once effects, which is the only kind of exactly-once that exists.

Q: How do we accept card numbers without every server in the company falling under PCI DSS audit?

Start with PSP-hosted fields: the checkout embeds the PSP's iframe, the PSP returns a token, and the token is all our API ever receives. This keeps us at the lowest PCI level and lets the whole payment pipeline run on ordinary infrastructure. Draw the vault on the diagram as the place where PANs would live if we ever need our own, and explain that it would be an isolated network with an HSM and its own audit trail. Moving to an own vault is a business decision about multi-PSP routing, not a technical necessity.

Q: Three systems have to agree on every payment. Why not wrap them in a distributed transaction?

Saga coordinated by the payment state machine, with the outbox making every step's event reliable and the ledger consuming those events idempotently. Compensation is explicit: a capture that fails after authorization voids the auth; a ledger posting that fails is retried from the event log, never skipped. The PSP is never inside our transaction; instead we record intent before calling it and resolve unknowns by querying it. Reconciliation runs nightly as the independent check. Be ready to say that at 100 TPS a single Postgres holding both payments and ledger is fine and that the separation is about access control and blast radius more than throughput.

Accept card payments for an e-commerce platform, move money correctly exactly once, and be able to prove it.

Last updated 2026-09-20. Difficulty: hard. Patterns: consistency, idempotency, ledger, fintech. Reported at Stripe and 5 more with Pro.

Walk through a strong candidate's answer, turn by turn.

The interviewer asks, the candidate answers and draws, and you press Next. Pause to answer yourself at the key decisions, and ask the coach anything along the way.

Walk through it Take the mock interview

Functional requirements

Accept a payment for an order. Card and wallet. Two phases: authorize (reserve funds) and capture (take them), possibly days apart.
Refunds, full and partial. Against a captured payment; multiple partial refunds must never exceed the captured amount.
Query payment status. Merchant and buyer can see: pending, authorized, captured, failed, refunded.
Idempotent retries. A client that times out and retries must never produce a second charge.
Merchant webhooks. Notify the merchant's system on every state change, with retries and ordering hints.
Merchant payouts / settlement. Aggregate captured funds minus fees and refunds, pay out on a schedule.
Reconciliation with the payment provider. Daily: every cent we think we moved matches what the PSP says it moved.
Out of scope. Fraud scoring, currency conversion, and being a card acquirer ourselves. We integrate with a PSP such as Stripe or Adyen.

Non-functional requirements

Correctness over everything (no double charge, no lost money). A duplicated or lost payment is a user-visible incident and a regulatory problem. Availability and latency are negotiable; this is not.
Strong consistency on the ledger (linearizable writes). Balances derived from the ledger must be exact at any point in time. No eventual consistency in money movement.
Authorization latency (p99 < 1 s). Dominated by the external PSP round trip (200–500 ms). Our own overhead budget is ~100 ms.
Availability of the auth path (99.999 %). Every minute down is lost revenue for every merchant. Degrade: queue captures and refunds, but keep authorizations flowing.
Auditability (immutable, append-only). Every state change is recorded with who, when, why. Nothing is ever updated in place; corrections are new entries.
PCI DSS scope minimized. Raw card numbers never touch our general infrastructure. A tiny tokenizing vault, or PSP-hosted fields, is the only PCI zone.
Scale (1 M payments/day). Modest by throughput standards. The difficulty is correctness under partial failure, not QPS.

Back-of-envelope estimates

Payments per second: ~12 avg · ~100 peak. 1 M/day ÷ 86 400 ≈ 11.6/s. Peaks (sales events, evening) run 5–10× the average. A single well-provisioned Postgres primary handles this with ease; throughput is not the design driver.
Ledger entries per payment: ~6–10 rows. Double entry: each movement is 2 rows. Authorize (hold) + capture + fee + payout allocation ≈ 4 movements = 8 rows. Refunds add 2–4 more. So ~10 M ledger rows/day.
Ledger storage growth: ~2 TB/year. 10 M rows/day × ~500 B/row ≈ 5 GB/day ≈ 1.8 TB/yr before indexes. Append-only, never deleted. Partition by month; archive old partitions to cold storage after the retention window.
PSP latency budget: 300–500 ms of the 1 s. External auth calls are the long pole. Our path (API → idempotency check → DB write → PSP call → DB write → response) must add under 100 ms, which means one or two DB round trips and no synchronous fan-out.
Webhook deliveries per day: ~4 M. Each payment emits ~3–4 state changes (created, authorized, captured, plus occasional refund). 1 M × 4 = 4 M webhooks/day, ~50/s. Retries with backoff for failing merchant endpoints roughly double the attempts.
Reconciliation batch: ~1 M records/day. One PSP settlement report per day, one row per transaction. Join against our ledger by PSP transaction id. Comfortably a single batch job; the interesting part is what to do with mismatches, not the volume.
Idempotency key store: ~1 M keys/day, 24 h TTL. One key per payment attempt, ~200 B each with the cached response. ~200 MB live at any time. Small enough for Redis, but it must be durable (Redis with AOF or a DB table), because losing keys re-enables double charges.

Components

Checkout client: Merchant web/app checkout. Collects card details inside a PSP-hosted iframe or our tokenizer so raw PANs never reach the merchant or our API. Generates an idempotency key per payment attempt and reuses it on every retry of that attempt.
Payment API (auth · validation · idempotency): Public API: create payment, capture, refund, get status. Authenticates the merchant, validates the request, and enforces idempotency before any side effect happens. Stateless.
Idempotency store (Redis (AOF) or Postgres table): Key: (merchant_id, idempotency_key). Value: request fingerprint, state (in_progress | done), and the final response. Prevents two concurrent or sequential requests with the same key from executing twice. Durable: losing this store reopens the double-charge hole.
Payment service (state machine): Owns the payment lifecycle: created → authorized → captured → refunded / failed. Every transition is a transactional write to the Payments DB plus an outbox row. Calls the PSP through the adapter and never trusts a timeout as a definitive outcome.
Payments DB (Postgres): Payment records, attempts, refunds, and the outbox table. Single-primary Postgres with synchronous replica for durability. Row-level locks on the payment row serialize concurrent transitions.
Ledger service (double-entry): The source of truth for money. Records every movement as balanced debit/credit entries between accounts (buyer receivable, merchant payable, fee revenue, PSP clearing). Append-only. Balances are computed from entries, not stored as mutable columns.
Ledger DB (Postgres · append-only): Journal and postings tables partitioned by month. Constraint: every journal entry's postings sum to zero. No UPDATE or DELETE grants for the application role. Periodic balance snapshots make "current balance" a cheap query.
Card vault / tokenizer (PCI zone): The only place raw card numbers exist, in an isolated network with its own audit and access controls. Swaps a PAN for an opaque token; the rest of the system only ever sees tokens. Alternatively outsourced entirely to PSP-hosted fields.
PSP adapter (Stripe / Adyen client): Wraps each provider behind one interface: authorize, capture, refund, void, get. Passes our idempotency key through to the PSP's own idempotency mechanism. Enforces timeouts, circuit breaking, and maps provider errors into our retryable / non-retryable / unknown classes.
External PSP (card networks behind it): Stripe, Adyen, Braintree. Talks to card networks and issuing banks. Slow (hundreds of ms), occasionally times out, and sends asynchronous webhooks for events we did not observe synchronously (disputes, delayed captures, bank-side reversals).
Event bus (Kafka · from outbox): Payment state-change events, published by an outbox relay so an event exists if and only if the DB transaction committed. Consumers: webhook dispatcher, ledger posting, analytics, fraud. Keyed by payment_id for per-payment ordering.
Webhook dispatcher: Consumes events and delivers signed HTTP callbacks to merchant endpoints. Retries with exponential backoff for up to days, records every attempt, and exposes a replay API. At-least-once: merchants must dedupe on event id.
Merchant backend: The merchant's own system. Receives webhooks to fulfil orders. Must verify the signature and dedupe on event id.
Reconciliation job (nightly batch): Pulls the PSP settlement report, joins it against our payments and ledger by PSP transaction id, and produces three lists: matched, in-ours-not-theirs, in-theirs-not-ours, plus amount mismatches. Opens cases; never auto-corrects money.
Payout service (scheduled): Computes each merchant's payable balance from the ledger (captured − refunds − fees − prior payouts), initiates a bank transfer via the PSP or a banking partner, and posts the payout to the ledger. Runs on a schedule with a reconciliation hold.

User flows

Card payment: authorize then capture. The main path. The buyer pays, we reserve funds at the PSP, record it durably, and later capture when the merchant ships. Every step is designed so that a crash anywhere leaves a recoverable state.
1. Client tokenizes the card in the vault; the API never sees the PAN. The card form posts directly to the vault (or the PSP's hosted fields). The client receives an opaque token. This keeps every other component out of PCI scope.
2. Client calls create-payment with the token, amount and a fresh idempotency key. The idempotency key is a UUID generated once per checkout attempt and stored client side. If the user clicks pay twice or the request times out, the same key is sent again.
3. API reserves the idempotency key: new → proceed, in-progress → 409, done → return cached response. An atomic SET NX with state=in_progress. The request body is fingerprinted and stored; a retry with the same key but a different amount is rejected as a client error rather than silently executed.
4. Payment service creates the payment row in state "created" together with an outbox event, in one transaction. We record intent before calling the PSP. If we crash after this step, a recovery worker finds "created" payments older than N seconds and resolves them by querying the PSP. Nothing is ever fire-and-forget.
5. Payment service calls the PSP adapter to authorize, passing the same idempotency key. The PSP call is the slow, external, unreliable step. The key is forwarded so the PSP also dedupes. Timeout 5 s; the outcome is one of success, definitive failure, or unknown (timeout / 5xx). Unknown is treated very differently from failure; see the state-machine deep dive.
6. On success, transition to "authorized" and store the PSP transaction id, again with an outbox event. The PSP transaction id is the join key for reconciliation. The row lock on the payment row means a concurrent webhook for the same payment waits and then sees the new state.
7. API marks the idempotency key done with the response and returns to the client. From now on any retry with this key returns exactly this response, without touching the Payment service.
8. Outbox relay publishes events; ledger posts the authorization hold and the webhook dispatcher notifies the merchant. Because the event came from the committed outbox row, it cannot exist without the state change and vice versa. Ledger posting is idempotent on event id. The merchant gets payment.authorized and can start fulfilment.
9. Later, the merchant calls capture; the same pattern runs: idempotency, state transition, PSP call, outbox, ledger, webhook. Capture moves the ledger from "authorized hold" to "merchant payable" and "fee revenue". The buyer's card is actually charged at this point. Authorizations expire after ~7 days if never captured.
Retry after a timeout with the same idempotency key. The scenario that separates a real payment system from a toy: the client timed out, does not know whether the charge happened, and retries. There must be exactly one charge and one consistent answer.
1. First attempt: API reserves the key and the Payment service calls the PSP, but the client's HTTP request times out while waiting. The client has no idea what happened. The server side is still running: the PSP may well succeed and the payment will become authorized.
2. Client retries with the same idempotency key. This is the contract: same attempt, same key. A new key would mean a new payment, and that is how double charges happen in naive systems.
3. API finds the key in state "in_progress" and returns 409 Conflict with retry-after. We do not execute a second payment and we do not block the request on the first one finishing. The client backs off and retries in a second or two.
4. Meanwhile the first attempt completes: state → authorized, key → done with the response. If instead the server itself had crashed mid-flight, the key would stay in_progress; a sweeper expires stale in_progress keys after the PSP timeout window and the recovery worker resolves the payment by querying the PSP with our idempotency key.
5. Client retries again; API returns the cached success response. One charge, one answer. The response is byte-identical to what the first attempt would have returned. The client cannot tell it ever timed out, which is exactly the point.
Partial refund. Refunds are payments in reverse with an extra invariant: the sum of refunds can never exceed the captured amount, even under concurrent requests.
1. Merchant requests a refund of 30 on a captured payment of 100, with an idempotency key. Refunds have their own keys because a merchant may legitimately issue several partial refunds on one payment.
2. Payment service locks the payment row and checks captured − refunded ≥ 30. SELECT … FOR UPDATE on the payment row serializes concurrent refunds. Two simultaneous 60-refunds on a 100 payment: the first succeeds, the second sees 40 remaining and is rejected. Without the lock both would pass the check.
3. Create the refund row in state "pending" plus an outbox event, then commit. Intent recorded before the external call, as always. Refunded-so-far is derived from refund rows in non-failed states, so a pending refund already counts against the limit.
4. Call the PSP to refund; on success transition refund → succeeded. Refunds at the PSP are often asynchronous: the API returns "pending" and a webhook arrives later with the final status. The state machine must accept both the synchronous and the webhook path idempotently.
5. Ledger posts the reversal entries; merchant is notified. Debit merchant payable 30, credit buyer receivable 30. Fees may or may not be refunded depending on the PSP; that is its own posting. The merchant's payable balance drops before the next payout is computed.
PSP webhook updates a payment asynchronously. The PSP tells us about things we did not observe synchronously: a delayed capture, a chargeback, a refund that finally settled. Webhooks arrive out of order, duplicated, and sometimes before our own transaction has committed.
1. PSP POSTs an event to our webhook endpoint. Verify the signature first; reject anything unsigned. Respond 200 quickly after durably enqueueing the event; process it asynchronously so a slow handler never causes the PSP to retry-storm us.
2. Look up the payment by PSP transaction id and check the event id has not been processed. A processed_psp_events table keyed by event id makes handling idempotent. PSPs explicitly deliver at-least-once.
3. Lock the payment row and apply the transition only if it is valid from the current state. Out-of-order example: "captured" arrives before "authorized". The state machine either accepts captured as implying authorized, or parks the event and re-applies it when the earlier one lands. Never apply a transition blindly. If the payment row does not exist yet (webhook beat our own commit), retry with backoff.
4. If the webhook contradicts our state (we think failed, PSP says captured) record it as a discrepancy instead of overwriting. Genuine conflicts are rare and always worth a human look. Overwriting silently would hide bugs and make reconciliation lie.
5. Commit with outbox event; ledger and merchant webhooks follow as usual. For a chargeback: ledger moves funds from merchant payable to a disputes account; the merchant gets payment.disputed and can submit evidence.
Nightly reconciliation finds a mismatch. Even with idempotency and outboxes, bugs, PSP incidents and network partitions produce drift. Reconciliation is the safety net that turns unknown unknowns into tickets.
1. Job downloads the PSP settlement report for the day. One row per transaction: PSP id, type, amount, currency, fee, status, timestamp. Usually a CSV or a paginated API. Reports for day D are typically final only on D+1 or D+2.
2. Join against our payments and refunds on PSP transaction id. Buckets: matched (same amount and status), missing-in-ours (PSP has a charge we do not), missing-in-theirs (we think we charged but PSP has nothing), amount or status mismatch.
3. Join our payments against the ledger: does every captured payment have balanced postings of the right amount? Internal consistency check, independent of the PSP. Catches consumer bugs where an event was published but the ledger posting failed or posted the wrong account.
4. Mismatch found: a payment we marked "failed" (PSP timeout) appears as captured at the PSP. Classic case: the PSP processed the auth but the response never reached us, and a bug treated the timeout as a failure instead of unknown. The buyer was charged; the merchant never shipped.
5. Job opens a case with both sides of the evidence; a human or a vetted auto-remediation resolves it. Remediation here: either void/refund at the PSP or transition the payment to captured and notify the merchant. The job itself never moves money automatically for mismatches. Metrics on mismatch rate per PSP per day are one of the most important dashboards in the system.
6. Payouts for the affected merchant are held until the case closes. A payout computed from a ledger with an open discrepancy could over- or under-pay. Holding is cheap; clawing back is not.

Deep dives

Idempotency key design

What exactly is stored under an idempotency key, for how long, and what happens on every kind of conflict?

The idempotency key is the single most important mechanism in the system: it is the only thing standing between a flaky network and a double charge. Its design has more corners than it looks. Who generates the key? What is its scope? What if the same key arrives with a different body? What if the first request is still running? What if it crashed halfway?

The store must be durable. A cache that can evict or lose keys silently re-enables the failure it exists to prevent. Redis is acceptable only with persistence and replication; many teams simply use a table in the payments database so the key and the payment commit together.

Client-generated key, scoped per merchant, durable store, response cached chosen
Server-derived key from request hash rejected
Idempotency only at the PSP layer rejected: as a second layer, never the only one

The answer: Client-generated UUID per attempt, scoped by (merchant_id, key), stored durably with the request fingerprint, a state (in_progress → done) and the final response, TTL 24 hours (long enough for any sane retry, short enough to bound storage). Reserve atomically before any side effect. Conflict rules: same key + same fingerprint + done → return cached response; same key + in_progress → 409 with retry-after; same key + different fingerprint → 422, never execute. Stale in_progress keys older than the PSP timeout are handed to a recovery worker that queries the PSP for the outcome. Forward the same key to the PSP as belt and braces.

The same idempotency key arrives with a different amount. What do you return?

A 422 or 409 with an explicit 'idempotency key reused with a different request' error, and you do not execute anything. This is almost always a client bug and silently executing either version would be wrong. Store a hash of the canonical request body alongside the key so this comparison is cheap.

Two requests with the same key arrive at exactly the same time on two API servers.

Both attempt the atomic reserve (SET NX or an INSERT with a unique constraint). Exactly one wins and proceeds; the other sees in_progress and returns 409 with retry-after. The atomicity of the store is what makes this safe, which is why it cannot be a local in-memory cache.

Why 24 hours? Why not keep keys forever?

Storage and semantics. Forever means the store grows without bound and a merchant reusing keys from a year ago gets confusing cached responses. 24 hours covers any realistic retry window with a large margin. Stripe keeps keys for 24 hours for the same reason. Payment records themselves are kept forever; only the retry-dedupe cache expires.

The API server crashes after reserving the key but before creating the payment. Now the key is stuck in_progress.

A sweeper expires in_progress keys older than the maximum end-to-end timeout, say 60 seconds. Because the payment row was never created, there was no PSP call, so the next retry safely starts over. If the crash happened after the payment row was created, the recovery worker for pending payments handles it independently of the key. The two mechanisms overlap on purpose.

What if the client does not send an idempotency key at all?

Reject the request for money-moving endpoints. Optional idempotency is the same as no idempotency in the failure case that matters. For a legacy client you can derive a key from (merchant, order id) as a stopgap, but document that it is weaker.

Double-entry ledger vs a balance column

Why not just keep a balance column per merchant and add or subtract from it?

A balance column is fast and obvious and it is how most systems start. It fails on three fronts: it has no memory (you cannot explain how the number got there), it is mutable (a bug or a race can corrupt it with no trace), and it cannot prove conservation (money that leaves one place must arrive somewhere else).

Double-entry bookkeeping is 500 years old because it solves exactly these problems. Every movement is a journal entry with two or more postings that sum to zero. Balances are derived by summing postings for an account. Nothing is ever edited; mistakes are corrected with new, reversing entries. The ledger becomes a complete, auditable history and the invariant "sum of all postings equals zero" is checkable at any time.

Append-only double-entry ledger, balances derived (with snapshots) chosen
Mutable balance column with a transaction log beside it situational: a denormalised read model on top of the ledger, never the source of truth
Balance column only rejected

The answer: Append-only journal in Postgres: journal_entries (id, event_id unique, created_at, description) and postings (entry_id, account_id, amount signed, currency). A deferred constraint checks that postings per entry sum to zero. The application role has INSERT and SELECT only. Balances come from a nightly snapshot table plus a sum of postings since the snapshot, which keeps reads fast without giving up derivation. Hot accounts (the platform fee account receives a posting on every payment) are the scaling concern: because postings are append-only there is no row contention, only index growth, and snapshots keep reads bounded. Idempotency on event_id means a replayed event cannot post twice.

The platform fee account receives a posting on every payment. How do you compute its balance without scanning millions of rows?

Balance snapshots: a job periodically writes (account, as_of, balance) and a balance read sums only postings after the latest snapshot. Reads stay bounded regardless of history. Because postings are append-only there is no write contention on a balance row, which is exactly the hot-row problem a mutable balance column would have.

How do you correct a mistake in the ledger, say a fee posted at the wrong rate?

Never edit. Post a reversing entry that undoes the original, then post the correct one, both referencing the original entry id and a reason. The history shows the mistake and the fix, which is what an auditor wants to see. Corrections are ordinary entries with a special type.

Multi-currency: a buyer pays in EUR, the merchant is paid in USD.

Every posting carries a currency, and an entry must balance per currency, not just in total. A conversion is modelled as two legs through an FX clearing account: EUR out of the buyer receivable into FX clearing, USD out of FX clearing into merchant payable, with the rate and timestamp recorded. FX gains and losses fall out as the residual in the clearing account.

Why keep a separate payments table at all? Isn't the ledger the source of truth?

The ledger is the source of truth for money; the payments table is the source of truth for workflow state: what the PSP said, what the merchant asked for, which webhooks were sent. Querying 'is this payment captured' from the ledger requires interpreting postings; from the payments table it is a column. They are reconciled against each other nightly, which is a feature: two independent views catch each other's bugs.

Payment state machine and the unknown outcome

The PSP call timed out. Did the charge happen? What do we do now?

Most bugs in payment systems come from collapsing three outcomes into two. A PSP call returns success, a definitive failure (card declined, invalid token), or an unknown: timeout, connection reset, 5xx, or the process died mid-call. Treating unknown as failure means retrying and double charging; treating it as success means shipping goods that were never paid for.

The state machine therefore has explicit uncertain states and every transition out of them is driven by evidence: a query to the PSP, a webhook, or the reconciliation report. Retrying an authorize blindly is never allowed.

Explicit "pending / unknown" state resolved by querying the PSP chosen
Retry the authorize on timeout situational: the PSP's idempotency guarantees are documented and tested, and the retry uses the identical key
Void on timeout, then start fresh situational: auth-only flows where a stale hold is worse than a retry
Treat timeout as failure rejected

The answer: States: created → authorizing → authorized → capturing → captured → (refunding → refunded) with failed reachable only from a definitive PSP failure and a terminal "requires_review" for contradictions. A timeout leaves the payment in authorizing. A recovery worker picks up payments in authorizing older than the PSP timeout and calls get-by-idempotency-key at the PSP: found and succeeded → authorized; found and failed → failed; not found → safe to retry the authorize with the same key, or fail after a bounded number of attempts. Every transition takes a row lock and validates the from-state, so a late webhook and the recovery worker cannot both apply. Time-in-authorizing is an alerting metric.

The PSP's status lookup also times out. Now what?

Stay in authorizing and retry the lookup with backoff for a bounded window, say 15 minutes. Lookups are safe to repeat indefinitely. If the window expires, move to requires_review and alert; do not fail the payment, because failing would tell the merchant not to ship when the buyer may have been charged. Reconciliation will settle it the next day if humans have not already.

A late webhook arrives saying 'authorized' after the recovery worker already marked the payment failed.

The transition failed to authorized is not valid, so the webhook handler records a discrepancy and alerts rather than applying it. This means our recovery logic and the PSP disagree, which is a bug or a PSP-side race, and a human should look. If it turns out the PSP is right, remediation is to move the payment to authorized manually or void it at the PSP.

How do you test the unknown-outcome paths? They never happen in a dev environment.

Fault injection in the PSP adapter: a test mode that returns timeouts, 5xx and success-without-response for specific test cards or amounts, plus a chaos setting that randomly injects them at a low rate in staging. Every state and transition in the machine should have a test that drives it. The recovery worker should be exercised continuously in staging, not only when something breaks.

What is the difference between a void and a refund, and when does the state machine use each?

Void cancels an authorization before capture; the hold disappears and no money moved, usually free. Refund returns captured funds; money moves back and fees are often not returned. The state machine voids from authorized and refunds from captured. Getting this wrong costs real money: refunding an uncaptured auth fails at the PSP, voiding a captured payment is not possible.

Why row locks on the payment instead of optimistic concurrency?

Both work. Row locks are simpler to reason about with several concurrent writers (API, webhook handler, recovery worker) and the contention per payment is tiny, so the cost is negligible. Optimistic concurrency with a version column is better if you must avoid holding a lock across the PSP call; but you should not be holding a lock across the PSP call in either design. Record intent, release, call the PSP, lock again to apply the result.

Reliable events: the transactional outbox

The payment row committed but the Kafka publish failed. Now the ledger and the merchant never hear about it. How do we make "state changed" and "event published" atomic?

Writing to a database and publishing to a message broker are two separate systems, so a crash between them leaves them inconsistent. Dual writes are one of the most common sources of silent data loss in event-driven systems. The fix is to make the event part of the database transaction and have a separate process move it to the broker.

With the outbox in place, every downstream consumer can rely on one property: an event exists if and only if the state change committed. Consumers still need idempotency, because the relay is at-least-once.

Transactional outbox table + relay (or CDC) chosen
Dual write: commit DB, then publish rejected
Two-phase commit across DB and broker rejected
Event sourcing: the event log is the database situational: greenfield with a team experienced in it; the ledger itself is effectively event-sourced

The answer: Every state transition inserts into an outbox table (event_id, aggregate_id, type, payload, created_at) in the same transaction as the payment row. A relay publishes unpublished rows to Kafka keyed by payment_id and marks them sent; CDC from the Postgres WAL is the higher-throughput version of the same idea. Consumers (ledger, webhooks, analytics) each track processed event ids so redelivery is harmless. This gives at-least-once delivery with exactly-once effects, which is the only kind of exactly-once that exists.

The outbox relay publishes an event, crashes before marking it sent, and publishes it again. What happens downstream?

Every consumer is idempotent on event id, so the ledger sees the id in its processed table and skips, and the webhook dispatcher does the same. At-least-once from the relay plus idempotent consumers gives exactly-once effects. This is why the event id is generated inside the original transaction, not by the relay.

How does the relay preserve ordering across payments and within a payment?

It reads the outbox in commit order (by an increasing id or the WAL position) and publishes to Kafka keyed by payment id, so all events for a payment land in one partition in order. Across payments there is no ordering requirement. With CDC the WAL gives you commit order for free.

What is the latency cost of the outbox, and does it matter here?

A polling relay adds its poll interval, typically 50 to 500 ms; CDC adds tens of ms. The synchronous API response does not wait for it, so the buyer sees no difference. The ledger and merchant webhook lag by that much, which is well within any reasonable expectation. If a consumer needed sub-10 ms it would need a different design, but nothing in a payment system does.

How big does the outbox table get and how do you keep it healthy?

Millions of rows a day. Delete or archive rows once published and older than a retention window, in batches to avoid long locks and bloat, or partition by day and drop partitions. Monitor the oldest unpublished row's age: that single metric tells you the relay is stuck before anyone notices missing webhooks.

PCI scope and card tokenization

How do we accept card numbers without every server in the company falling under PCI DSS audit?

Any system component that stores, processes or transmits a primary account number is in PCI scope, and scope is expensive: quarterly scans, network segmentation, access reviews, and an annual audit for every in-scope host. The design goal is to make the in-scope footprint as small as possible, ideally zero.

Tokenization is the mechanism: swap the PAN for a random token at the earliest possible moment, and let only the token flow through the system. The token is useless outside the vault that issued it.

PSP-hosted fields / client-side tokenization chosen
Own card vault with a tokenizer in an isolated PCI zone situational: multi-PSP routing at large scale, or when vendor lock-in is a strategic risk
Encrypt PANs in the main database rejected

The answer: Start with PSP-hosted fields: the checkout embeds the PSP's iframe, the PSP returns a token, and the token is all our API ever receives. This keeps us at the lowest PCI level and lets the whole payment pipeline run on ordinary infrastructure. Draw the vault on the diagram as the place where PANs would live if we ever need our own, and explain that it would be an isolated network with an HSM and its own audit trail. Moving to an own vault is a business decision about multi-PSP routing, not a technical necessity.

With PSP-hosted fields, how do you support saving a card for later or one-click checkout?

The PSP returns a reusable payment method token tied to the customer, which we store in our database as an opaque string. Charging it later is an API call with that token. The card number never enters our systems; the token is useless to anyone without our PSP API key. Card updater services at the PSP keep it valid when the card is reissued.

We want to switch from Stripe to Adyen. What happens to saved cards?

Tokens are PSP-specific, so this is the lock-in cost. Options: a PCI-compliant token migration where the old PSP transfers card data directly to the new one, which most large PSPs support with paperwork and time; or run both and re-collect cards on next purchase. An own vault avoids this entirely, which is the main business argument for building one.

What exactly is still in PCI scope under this design?

The checkout page that embeds the iframe, because it could be modified to skim, so it needs integrity controls and is covered under SAQ-A or SAQ-A-EP depending on how the fields are integrated. Our servers, database and network are out of scope because no PAN ever transits them. Logging discipline still matters: never log request bodies from the checkout path in case a client sends a raw PAN by mistake.

If you did build the vault, what are the three things you would get right first?

Network isolation with no inbound access except the tokenize and detokenize endpoints; encryption with keys in an HSM and envelope encryption so the data key never leaves the HSM in plaintext; and an audit trail on every detokenize call with tight authorization, since detokenize is the operation an attacker wants. Everything else follows from the standard.

Consistency across PSP, payments DB and ledger

Three systems have to agree on every payment. Why not wrap them in a distributed transaction?

The PSP is an external party over HTTP; it cannot join our transaction. The payments DB and ledger DB could in principle share a transaction, but coupling them means the ledger's availability gates every authorization and vice versa. So we have three systems that cannot be atomically updated together and must nonetheless never disagree about money.

The answer is a saga: a sequence of local transactions, each with a compensating action, driven by durable state and events, plus reconciliation to catch what the saga logic missed. The payments DB is the coordinator's state; the ledger is eventually consistent with it by seconds; and the PSP is treated as an external source of truth that we periodically re-verify.

Saga with outbox events + nightly reconciliation chosen
Two-phase commit across our databases rejected
Single database for payments and ledger, PSP outside situational: early stage; entirely defensible at 1 M payments/day

The answer: Saga coordinated by the payment state machine, with the outbox making every step's event reliable and the ledger consuming those events idempotently. Compensation is explicit: a capture that fails after authorization voids the auth; a ledger posting that fails is retried from the event log, never skipped. The PSP is never inside our transaction; instead we record intent before calling it and resolve unknowns by querying it. Reconciliation runs nightly as the independent check. Be ready to say that at 100 TPS a single Postgres holding both payments and ledger is fine and that the separation is about access control and blast radius more than throughput.

A capture succeeds at the PSP but our database write of 'captured' fails. Now the PSP has our money and we think it is only authorized.

The payment stays in capturing, which is an uncertain state. The recovery worker queries the PSP, finds the capture, and applies the transition. If the DB is down entirely, nothing progresses until it is back, which is correct: we never claim a state we did not durably record. Reconciliation catches any case the worker misses.

Why is the ledger allowed to lag the payments DB? Isn't strong consistency the requirement?

Strong consistency applies within the ledger: its writes are linearizable and its balances are exact for what it has recorded. Between the payments DB and the ledger there is a bounded lag of the outbox relay, during which a payout computed from the ledger could miss a very recent capture. That is handled by the payout service applying a settlement delay, typically days, so the lag is irrelevant. Consistency between systems is achieved by ordering plus delay plus reconciliation, not by transactions.

Convince me 2PC would not work even if the PSP supported it.

The coordinator must hold both participants in a prepared state until the decision, so a slow PSP blocks a row in our database and a coordinator crash leaves in-doubt transactions that block until recovery. The availability of the whole system becomes the product of all participants' availability. Payment systems care about the authorization path being up more than about a few seconds of ledger lag, so the tradeoff is backwards.

What is the compensating action if the ledger posting is rejected, say the entry does not balance?

That is a bug, not a business failure, so there is no compensation: the event goes to a dead-letter queue, an alert fires, and the payment is flagged so payouts for that merchant are held. Compensations are for business failures like a capture declined after authorization. A non-balancing entry means code is wrong and money should stop moving for that account until a human looks.

How would you shard this if it were 100x bigger?

Shard the payments DB by merchant id, since a payment never crosses merchants and merchant-level queries stay local. The ledger shards by account, with cross-shard entries handled by a two-step posting through a clearing account per shard pair, which keeps each shard's entries balanced. The outbox and Kafka partitioning follow the same key. At that scale you would also give the idempotency store its own cluster keyed the same way.