Design a Payment System

• Accept a payment for an order — Card and wallet. Two phases: authorize (reserve funds) and capture (take them), possibly days apart.
• Refunds, full and partial — Against a captured payment; multiple partial refunds must never exceed the captured amount.
• Query payment status — Merchant and buyer can see: pending, authorized, captured, failed, refunded.
• Idempotent retries — A client that times out and retries must never produce a second charge.
• Merchant webhooks — Notify the merchant's system on every state change, with retries and ordering hints.
• Merchant payouts / settlement — Aggregate captured funds minus fees and refunds, pay out on a schedule.
• Reconciliation with the payment provider — Daily: every cent we think we moved matches what the PSP says it moved.
• Out of scope — Fraud scoring, currency conversion, and being a card acquirer ourselves. We integrate with a PSP such as Stripe or Adyen.

• Correctness over everything (no double charge, no lost money) — A duplicated or lost payment is a user-visible incident and a regulatory problem. Availability and latency are negotiable; this is not.
• Strong consistency on the ledger (linearizable writes) — Balances derived from the ledger must be exact at any point in time. No eventual consistency in money movement.
• Authorization latency (p99 < 1 s) — Dominated by the external PSP round trip (200–500 ms). Our own overhead budget is ~100 ms.
• Availability of the auth path (99.999 %) — Every minute down is lost revenue for every merchant. Degrade: queue captures and refunds, but keep authorizations flowing.
• Auditability (immutable, append-only) — Every state change is recorded with who, when, why. Nothing is ever updated in place; corrections are new entries.
• PCI DSS scope minimized — Raw card numbers never touch our general infrastructure. A tiny tokenizing vault, or PSP-hosted fields, is the only PCI zone.
• Scale (1 M payments/day) — Modest by throughput standards. The difficulty is correctness under partial failure, not QPS.

• Payments per second: ~12 avg · ~100 peak — 1 M/day ÷ 86 400 ≈ 11.6/s. Peaks (sales events, evening) run 5–10× the average. A single well-provisioned Postgres primary handles this with ease; throughput is not the design driver.
• Ledger entries per payment: ~6–10 rows — Double entry: each movement is 2 rows. Authorize (hold) + capture + fee + payout allocation ≈ 4 movements = 8 rows. Refunds add 2–4 more. So ~10 M ledger rows/day.
• Ledger storage growth: ~2 TB/year — 10 M rows/day × ~500 B/row ≈ 5 GB/day ≈ 1.8 TB/yr before indexes. Append-only, never deleted. Partition by month; archive old partitions to cold storage after the retention window.
• PSP latency budget: 300–500 ms of the 1 s — External auth calls are the long pole. Our path (API → idempotency check → DB write → PSP call → DB write → response) must add under 100 ms, which means one or two DB round trips and no synchronous fan-out.
• Webhook deliveries per day: ~4 M — Each payment emits ~3–4 state changes (created, authorized, captured, plus occasional refund). 1 M × 4 = 4 M webhooks/day, ~50/s. Retries with backoff for failing merchant endpoints roughly double the attempts.
• Reconciliation batch: ~1 M records/day — One PSP settlement report per day, one row per transaction. Join against our ledger by PSP transaction id. Comfortably a single batch job; the interesting part is what to do with mismatches, not the volume.
• Idempotency key store: ~1 M keys/day, 24 h TTL — One key per payment attempt, ~200 B each with the cached response. ~200 MB live at any time. Small enough for Redis, but it must be durable (Redis with AOF or a DB table), because losing keys re-enables double charges.

• Checkout client — Merchant web/app checkout. Collects card details inside a PSP-hosted iframe or our tokenizer so raw PANs never reach the merchant or our API. Generates an idempotency key per payment attempt and reuses it on every retry of that attempt.
• Payment API (auth · validation · idempotency) — Public API: create payment, capture, refund, get status. Authenticates the merchant, validates the request, and enforces idempotency before any side effect happens. Stateless.
• Idempotency store (Redis (AOF) or Postgres table) — Key: (merchant_id, idempotency_key). Value: request fingerprint, state (in_progress | done), and the final response. Prevents two concurrent or sequential requests with the same key from executing twice. Durable: losing this store reopens the double-charge hole.
• Payment service (state machine) — Owns the payment lifecycle: created → authorized → captured → refunded / failed. Every transition is a transactional write to the Payments DB plus an outbox row. Calls the PSP through the adapter and never trusts a timeout as a definitive outcome.
• Payments DB (Postgres) — Payment records, attempts, refunds, and the outbox table. Single-primary Postgres with synchronous replica for durability. Row-level locks on the payment row serialize concurrent transitions.
• Ledger service (double-entry) — The source of truth for money. Records every movement as balanced debit/credit entries between accounts (buyer receivable, merchant payable, fee revenue, PSP clearing). Append-only. Balances are computed from entries, not stored as mutable columns.
• Ledger DB (Postgres · append-only) — Journal and postings tables partitioned by month. Constraint: every journal entry's postings sum to zero. No UPDATE or DELETE grants for the application role. Periodic balance snapshots make "current balance" a cheap query.
• Card vault / tokenizer (PCI zone) — The only place raw card numbers exist, in an isolated network with its own audit and access controls. Swaps a PAN for an opaque token; the rest of the system only ever sees tokens. Alternatively outsourced entirely to PSP-hosted fields.
• PSP adapter (Stripe / Adyen client) — Wraps each provider behind one interface: authorize, capture, refund, void, get. Passes our idempotency key through to the PSP's own idempotency mechanism. Enforces timeouts, circuit breaking, and maps provider errors into our retryable / non-retryable / unknown classes.
• External PSP (card networks behind it) — Stripe, Adyen, Braintree. Talks to card networks and issuing banks. Slow (hundreds of ms), occasionally times out, and sends asynchronous webhooks for events we did not observe synchronously (disputes, delayed captures, bank-side reversals).
• Event bus (Kafka · from outbox) — Payment state-change events, published by an outbox relay so an event exists if and only if the DB transaction committed. Consumers: webhook dispatcher, ledger posting, analytics, fraud. Keyed by payment_id for per-payment ordering.
• Webhook dispatcher — Consumes events and delivers signed HTTP callbacks to merchant endpoints. Retries with exponential backoff for up to days, records every attempt, and exposes a replay API. At-least-once: merchants must dedupe on event id.
• Merchant backend — The merchant's own system. Receives webhooks to fulfil orders. Must verify the signature and dedupe on event id.
• Reconciliation job (nightly batch) — Pulls the PSP settlement report, joins it against our payments and ledger by PSP transaction id, and produces three lists: matched, in-ours-not-theirs, in-theirs-not-ours, plus amount mismatches. Opens cases; never auto-corrects money.
• Payout service (scheduled) — Computes each merchant's payable balance from the ledger (captured − refunds − fees − prior payouts), initiates a bank transfer via the PSP or a banking partner, and posts the payout to the ledger. Runs on a schedule with a reconciliation hold.

1. Card payment: authorize then capture — The main path. The buyer pays, we reserve funds at the PSP, record it durably, and later capture when the merchant ships. Every step is designed so that a crash anywhere leaves a recoverable state.
   1. Client tokenizes the card in the vault; the API never sees the PAN.
   2. Client calls create-payment with the token, amount and a fresh idempotency key.
   3. API reserves the idempotency key: new → proceed, in-progress → 409, done → return cached response.
   4. Payment service creates the payment row in state "created" together with an outbox event, in one transaction.
   5. Payment service calls the PSP adapter to authorize, passing the same idempotency key.
   6. On success, transition to "authorized" and store the PSP transaction id, again with an outbox event.
   7. API marks the idempotency key done with the response and returns to the client.
   8. Outbox relay publishes events; ledger posts the authorization hold and the webhook dispatcher notifies the merchant.
   9. Later, the merchant calls capture; the same pattern runs: idempotency, state transition, PSP call, outbox, ledger, webhook.
2. Retry after a timeout with the same idempotency key — The scenario that separates a real payment system from a toy: the client timed out, does not know whether the charge happened, and retries. There must be exactly one charge and one consistent answer.
   1. First attempt: API reserves the key and the Payment service calls the PSP, but the client's HTTP request times out while waiting.
   2. Client retries with the same idempotency key.
   3. API finds the key in state "in_progress" and returns 409 Conflict with retry-after.
   4. Meanwhile the first attempt completes: state → authorized, key → done with the response.
   5. Client retries again; API returns the cached success response. One charge, one answer.
3. Partial refund — Refunds are payments in reverse with an extra invariant: the sum of refunds can never exceed the captured amount, even under concurrent requests.
   1. Merchant requests a refund of 30 on a captured payment of 100, with an idempotency key.
   2. Payment service locks the payment row and checks captured − refunded ≥ 30.
   3. Create the refund row in state "pending" plus an outbox event, then commit.
   4. Call the PSP to refund; on success transition refund → succeeded.
   5. Ledger posts the reversal entries; merchant is notified.
4. PSP webhook updates a payment asynchronously — The PSP tells us about things we did not observe synchronously: a delayed capture, a chargeback, a refund that finally settled. Webhooks arrive out of order, duplicated, and sometimes before our own transaction has committed.
   1. PSP POSTs an event to our webhook endpoint.
   2. Look up the payment by PSP transaction id and check the event id has not been processed.
   3. Lock the payment row and apply the transition only if it is valid from the current state.
   4. If the webhook contradicts our state (we think failed, PSP says captured) record it as a discrepancy instead of overwriting.
   5. Commit with outbox event; ledger and merchant webhooks follow as usual.
5. Nightly reconciliation finds a mismatch — Even with idempotency and outboxes, bugs, PSP incidents and network partitions produce drift. Reconciliation is the safety net that turns unknown unknowns into tickets.
   1. Job downloads the PSP settlement report for the day.
   2. Join against our payments and refunds on PSP transaction id.
   3. Join our payments against the ledger: does every captured payment have balanced postings of the right amount?
   4. Mismatch found: a payment we marked "failed" (PSP timeout) appears as captured at the PSP.
   5. Job opens a case with both sides of the evidence; a human or a vetted auto-remediation resolves it.
   6. Payouts for the affected merchant are held until the case closes.

Ask: What exactly is stored under an idempotency key, for how long, and what happens on every kind of conflict?

Good answers name: Client-generated key, scoped per merchant, durable store, response cached, Server-derived key from request hash, Idempotency only at the PSP layer.

Our pick: Client-generated UUID per attempt, scoped by (merchant_id, key), stored durably with the request fingerprint, a state (in_progress → done) and the final response, TTL 24 hours (long enough for any sane retry, short enough to bound storage). Reserve atomically before any side effect. Conflict rules: same key + same fingerprint + done → return cached response; same key + in_progress → 409 with retry-after; same key + different fingerprint → 422, never execute. Stale in_progress keys older than the PSP timeout are handed to a recovery worker that queries the PSP for the outcome. Forward the same key to the PSP as belt and braces.

1. The same idempotency key arrives with a different amount. What do you return?
   A 422 or 409 with an explicit 'idempotency key reused with a different request' error, and you do not execute anything. This is almost always a client bug and silently executing either version would be wrong. Store a hash of the canonical request body alongside the key so this comparison is cheap.
2. Two requests with the same key arrive at exactly the same time on two API servers.
   Both attempt the atomic reserve (SET NX or an INSERT with a unique constraint). Exactly one wins and proceeds; the other sees in_progress and returns 409 with retry-after. The atomicity of the store is what makes this safe, which is why it cannot be a local in-memory cache.
3. Why 24 hours? Why not keep keys forever?
   Storage and semantics. Forever means the store grows without bound and a merchant reusing keys from a year ago gets confusing cached responses. 24 hours covers any realistic retry window with a large margin. Stripe keeps keys for 24 hours for the same reason. Payment records themselves are kept forever; only the retry-dedupe cache expires.
4. The API server crashes after reserving the key but before creating the payment. Now the key is stuck in_progress.
   A sweeper expires in_progress keys older than the maximum end-to-end timeout, say 60 seconds. Because the payment row was never created, there was no PSP call, so the next retry safely starts over. If the crash happened after the payment row was created, the recovery worker for pending payments handles it independently of the key. The two mechanisms overlap on purpose.
5. What if the client does not send an idempotency key at all?
   Reject the request for money-moving endpoints. Optional idempotency is the same as no idempotency in the failure case that matters. For a legacy client you can derive a key from (merchant, order id) as a stopgap, but document that it is weaker.

Ask: Why not just keep a balance column per merchant and add or subtract from it?

Good answers name: Append-only double-entry ledger, balances derived (with snapshots), Mutable balance column with a transaction log beside it, Balance column only.

Our pick: Append-only journal in Postgres: journal_entries (id, event_id unique, created_at, description) and postings (entry_id, account_id, amount signed, currency). A deferred constraint checks that postings per entry sum to zero. The application role has INSERT and SELECT only. Balances come from a nightly snapshot table plus a sum of postings since the snapshot, which keeps reads fast without giving up derivation. Hot accounts (the platform fee account receives a posting on every payment) are the scaling concern: because postings are append-only there is no row contention, only index growth, and snapshots keep reads bounded. Idempotency on event_id means a replayed event cannot post twice.

1. The platform fee account receives a posting on every payment. How do you compute its balance without scanning millions of rows?
   Balance snapshots: a job periodically writes (account, as_of, balance) and a balance read sums only postings after the latest snapshot. Reads stay bounded regardless of history. Because postings are append-only there is no write contention on a balance row, which is exactly the hot-row problem a mutable balance column would have.
2. How do you correct a mistake in the ledger, say a fee posted at the wrong rate?
   Never edit. Post a reversing entry that undoes the original, then post the correct one, both referencing the original entry id and a reason. The history shows the mistake and the fix, which is what an auditor wants to see. Corrections are ordinary entries with a special type.
3. Multi-currency: a buyer pays in EUR, the merchant is paid in USD.
   Every posting carries a currency, and an entry must balance per currency, not just in total. A conversion is modelled as two legs through an FX clearing account: EUR out of the buyer receivable into FX clearing, USD out of FX clearing into merchant payable, with the rate and timestamp recorded. FX gains and losses fall out as the residual in the clearing account.
4. Why keep a separate payments table at all? Isn't the ledger the source of truth?
   The ledger is the source of truth for money; the payments table is the source of truth for workflow state: what the PSP said, what the merchant asked for, which webhooks were sent. Querying 'is this payment captured' from the ledger requires interpreting postings; from the payments table it is a column. They are reconciled against each other nightly, which is a feature: two independent views catch each other's bugs.

Ask: The PSP call timed out. Did the charge happen? What do we do now?

Good answers name: Explicit "pending / unknown" state resolved by querying the PSP, Retry the authorize on timeout, Void on timeout, then start fresh, Treat timeout as failure.

Our pick: States: created → authorizing → authorized → capturing → captured → (refunding → refunded) with failed reachable only from a definitive PSP failure and a terminal "requires_review" for contradictions. A timeout leaves the payment in authorizing. A recovery worker picks up payments in authorizing older than the PSP timeout and calls get-by-idempotency-key at the PSP: found and succeeded → authorized; found and failed → failed; not found → safe to retry the authorize with the same key, or fail after a bounded number of attempts. Every transition takes a row lock and validates the from-state, so a late webhook and the recovery worker cannot both apply. Time-in-authorizing is an alerting metric.

1. The PSP's status lookup also times out. Now what?
   Stay in authorizing and retry the lookup with backoff for a bounded window, say 15 minutes. Lookups are safe to repeat indefinitely. If the window expires, move to requires_review and alert; do not fail the payment, because failing would tell the merchant not to ship when the buyer may have been charged. Reconciliation will settle it the next day if humans have not already.
2. A late webhook arrives saying 'authorized' after the recovery worker already marked the payment failed.
   The transition failed to authorized is not valid, so the webhook handler records a discrepancy and alerts rather than applying it. This means our recovery logic and the PSP disagree, which is a bug or a PSP-side race, and a human should look. If it turns out the PSP is right, remediation is to move the payment to authorized manually or void it at the PSP.
3. How do you test the unknown-outcome paths? They never happen in a dev environment.
   Fault injection in the PSP adapter: a test mode that returns timeouts, 5xx and success-without-response for specific test cards or amounts, plus a chaos setting that randomly injects them at a low rate in staging. Every state and transition in the machine should have a test that drives it. The recovery worker should be exercised continuously in staging, not only when something breaks.
4. What is the difference between a void and a refund, and when does the state machine use each?
   Void cancels an authorization before capture; the hold disappears and no money moved, usually free. Refund returns captured funds; money moves back and fees are often not returned. The state machine voids from authorized and refunds from captured. Getting this wrong costs real money: refunding an uncaptured auth fails at the PSP, voiding a captured payment is not possible.
5. Why row locks on the payment instead of optimistic concurrency?
   Both work. Row locks are simpler to reason about with several concurrent writers (API, webhook handler, recovery worker) and the contention per payment is tiny, so the cost is negligible. Optimistic concurrency with a version column is better if you must avoid holding a lock across the PSP call; but you should not be holding a lock across the PSP call in either design. Record intent, release, call the PSP, lock again to apply the result.

Ask: The payment row committed but the Kafka publish failed. Now the ledger and the merchant never hear about it. How do we make "state changed" and "event published" atomic?

Good answers name: Transactional outbox table + relay (or CDC), Dual write: commit DB, then publish, Two-phase commit across DB and broker, Event sourcing: the event log is the database.

Our pick: Every state transition inserts into an outbox table (event_id, aggregate_id, type, payload, created_at) in the same transaction as the payment row. A relay publishes unpublished rows to Kafka keyed by payment_id and marks them sent; CDC from the Postgres WAL is the higher-throughput version of the same idea. Consumers (ledger, webhooks, analytics) each track processed event ids so redelivery is harmless. This gives at-least-once delivery with exactly-once effects, which is the only kind of exactly-once that exists.

1. The outbox relay publishes an event, crashes before marking it sent, and publishes it again. What happens downstream?
   Every consumer is idempotent on event id, so the ledger sees the id in its processed table and skips, and the webhook dispatcher does the same. At-least-once from the relay plus idempotent consumers gives exactly-once effects. This is why the event id is generated inside the original transaction, not by the relay.
2. How does the relay preserve ordering across payments and within a payment?
   It reads the outbox in commit order (by an increasing id or the WAL position) and publishes to Kafka keyed by payment id, so all events for a payment land in one partition in order. Across payments there is no ordering requirement. With CDC the WAL gives you commit order for free.
3. What is the latency cost of the outbox, and does it matter here?
   A polling relay adds its poll interval, typically 50 to 500 ms; CDC adds tens of ms. The synchronous API response does not wait for it, so the buyer sees no difference. The ledger and merchant webhook lag by that much, which is well within any reasonable expectation. If a consumer needed sub-10 ms it would need a different design, but nothing in a payment system does.
4. How big does the outbox table get and how do you keep it healthy?
   Millions of rows a day. Delete or archive rows once published and older than a retention window, in batches to avoid long locks and bloat, or partition by day and drop partitions. Monitor the oldest unpublished row's age: that single metric tells you the relay is stuck before anyone notices missing webhooks.

Ask: How do we accept card numbers without every server in the company falling under PCI DSS audit?

Good answers name: PSP-hosted fields / client-side tokenization, Own card vault with a tokenizer in an isolated PCI zone, Encrypt PANs in the main database.

Our pick: Start with PSP-hosted fields: the checkout embeds the PSP's iframe, the PSP returns a token, and the token is all our API ever receives. This keeps us at the lowest PCI level and lets the whole payment pipeline run on ordinary infrastructure. Draw the vault on the diagram as the place where PANs would live if we ever need our own, and explain that it would be an isolated network with an HSM and its own audit trail. Moving to an own vault is a business decision about multi-PSP routing, not a technical necessity.

1. With PSP-hosted fields, how do you support saving a card for later or one-click checkout?
   The PSP returns a reusable payment method token tied to the customer, which we store in our database as an opaque string. Charging it later is an API call with that token. The card number never enters our systems; the token is useless to anyone without our PSP API key. Card updater services at the PSP keep it valid when the card is reissued.
2. We want to switch from Stripe to Adyen. What happens to saved cards?
   Tokens are PSP-specific, so this is the lock-in cost. Options: a PCI-compliant token migration where the old PSP transfers card data directly to the new one, which most large PSPs support with paperwork and time; or run both and re-collect cards on next purchase. An own vault avoids this entirely, which is the main business argument for building one.
3. What exactly is still in PCI scope under this design?
   The checkout page that embeds the iframe, because it could be modified to skim, so it needs integrity controls and is covered under SAQ-A or SAQ-A-EP depending on how the fields are integrated. Our servers, database and network are out of scope because no PAN ever transits them. Logging discipline still matters: never log request bodies from the checkout path in case a client sends a raw PAN by mistake.
4. If you did build the vault, what are the three things you would get right first?
   Network isolation with no inbound access except the tokenize and detokenize endpoints; encryption with keys in an HSM and envelope encryption so the data key never leaves the HSM in plaintext; and an audit trail on every detokenize call with tight authorization, since detokenize is the operation an attacker wants. Everything else follows from the standard.

Ask: Three systems have to agree on every payment. Why not wrap them in a distributed transaction?

Good answers name: Saga with outbox events + nightly reconciliation, Two-phase commit across our databases, Single database for payments and ledger, PSP outside.

Our pick: Saga coordinated by the payment state machine, with the outbox making every step's event reliable and the ledger consuming those events idempotently. Compensation is explicit: a capture that fails after authorization voids the auth; a ledger posting that fails is retried from the event log, never skipped. The PSP is never inside our transaction; instead we record intent before calling it and resolve unknowns by querying it. Reconciliation runs nightly as the independent check. Be ready to say that at 100 TPS a single Postgres holding both payments and ledger is fine and that the separation is about access control and blast radius more than throughput.

1. A capture succeeds at the PSP but our database write of 'captured' fails. Now the PSP has our money and we think it is only authorized.
   The payment stays in capturing, which is an uncertain state. The recovery worker queries the PSP, finds the capture, and applies the transition. If the DB is down entirely, nothing progresses until it is back, which is correct: we never claim a state we did not durably record. Reconciliation catches any case the worker misses.
2. Why is the ledger allowed to lag the payments DB? Isn't strong consistency the requirement?
   Strong consistency applies within the ledger: its writes are linearizable and its balances are exact for what it has recorded. Between the payments DB and the ledger there is a bounded lag of the outbox relay, during which a payout computed from the ledger could miss a very recent capture. That is handled by the payout service applying a settlement delay, typically days, so the lag is irrelevant. Consistency between systems is achieved by ordering plus delay plus reconciliation, not by transactions.
3. Convince me 2PC would not work even if the PSP supported it.
   The coordinator must hold both participants in a prepared state until the decision, so a slow PSP blocks a row in our database and a coordinator crash leaves in-doubt transactions that block until recovery. The availability of the whole system becomes the product of all participants' availability. Payment systems care about the authorization path being up more than about a few seconds of ledger lag, so the tradeoff is backwards.
4. What is the compensating action if the ledger posting is rejected, say the entry does not balance?
   That is a bug, not a business failure, so there is no compensation: the event goes to a dead-letter queue, an alert fires, and the payment is flagged so payouts for that merchant are held. Compensations are for business failures like a capture declined after authorization. A non-balancing entry means code is wrong and money should stop moving for that account until a human looks.
5. How would you shard this if it were 100x bigger?
   Shard the payments DB by merchant id, since a payment never crosses merchants and merchant-level queries stay local. The ledger shards by account, with cross-shard entries handled by a two-step posting through a clearing account per shard pair, which keeps each shard's entries balanced. The outbox and Kafka partitioning follow the same key. At that scale you would also give the idempotency store its own cluster keyed the same way.