Design a Payment System

Accept card payments for an e-commerce platform, move money correctly exactly once, and be able to prove it.

Difficulty: hard. Patterns: consistency, idempotency, ledger, fintech. Reported at Stripe, Amazon, PayPal, Uber, Airbnb, Shopify.

Study shows every answer; Practice hides them until you have produced your own.

Functional requirements

• Accept a payment for an order. Card and wallet. Two phases: authorize (reserve funds) and capture (take them), possibly days apart.
• Refunds, full and partial. Against a captured payment; multiple partial refunds must never exceed the captured amount.
• Query payment status. Merchant and buyer can see: pending, authorized, captured, failed, refunded.
• Idempotent retries. A client that times out and retries must never produce a second charge.
• Merchant webhooks. Notify the merchant's system on every state change, with retries and ordering hints.
• Merchant payouts / settlement. Aggregate captured funds minus fees and refunds, pay out on a schedule.
• Reconciliation with the payment provider. Daily: every cent we think we moved matches what the PSP says it moved.
• Out of scope. Fraud scoring, currency conversion, and being a card acquirer ourselves. We integrate with a PSP such as Stripe or Adyen.

Non-functional requirements

• Correctness over everything (no double charge, no lost money). A duplicated or lost payment is a user-visible incident and a regulatory problem. Availability and latency are negotiable; this is not.
• Strong consistency on the ledger (linearizable writes). Balances derived from the ledger must be exact at any point in time. No eventual consistency in money movement.
• Authorization latency (p99 < 1 s). Dominated by the external PSP round trip (200–500 ms). Our own overhead budget is ~100 ms.
• Availability of the auth path (99.999 %). Every minute down is lost revenue for every merchant. Degrade: queue captures and refunds, but keep authorizations flowing.
• Auditability (immutable, append-only). Every state change is recorded with who, when, why. Nothing is ever updated in place; corrections are new entries.
• PCI DSS scope minimized. Raw card numbers never touch our general infrastructure. A tiny tokenizing vault, or PSP-hosted fields, is the only PCI zone.
• Scale (1 M payments/day). Modest by throughput standards. The difficulty is correctness under partial failure, not QPS.

Back-of-envelope estimates

• Payments per second: ~12 avg · ~100 peak. 1 M/day ÷ 86 400 ≈ 11.6/s. Peaks (sales events, evening) run 5–10× the average. A single well-provisioned Postgres primary handles this with ease; throughput is not the design driver.
• Ledger entries per payment: ~6–10 rows. Double entry: each movement is 2 rows. Authorize (hold) + capture + fee + payout allocation ≈ 4 movements = 8 rows. Refunds add 2–4 more. So ~10 M ledger rows/day.
• Ledger storage growth: ~2 TB/year. 10 M rows/day × ~500 B/row ≈ 5 GB/day ≈ 1.8 TB/yr before indexes. Append-only, never deleted. Partition by month; archive old partitions to cold storage after the retention window.
• PSP latency budget: 300–500 ms of the 1 s. External auth calls are the long pole. Our path (API → idempotency check → DB write → PSP call → DB write → response) must add under 100 ms, which means one or two DB round trips and no synchronous fan-out.
• Webhook deliveries per day: ~4 M. Each payment emits ~3–4 state changes (created, authorized, captured, plus occasional refund). 1 M × 4 = 4 M webhooks/day, ~50/s. Retries with backoff for failing merchant endpoints roughly double the attempts.
• Reconciliation batch: ~1 M records/day. One PSP settlement report per day, one row per transaction. Join against our ledger by PSP transaction id. Comfortably a single batch job; the interesting part is what to do with mismatches, not the volume.
• Idempotency key store: ~1 M keys/day, 24 h TTL. One key per payment attempt, ~200 B each with the cached response. ~200 MB live at any time. Small enough for Redis, but it must be durable (Redis with AOF or a DB table), because losing keys re-enables double charges.

Components

• Checkout client: Merchant web/app checkout. Collects card details inside a PSP-hosted iframe or our tokenizer so raw PANs never reach the merchant or our API. Generates an idempotency key per payment attempt and reuses it on every retry of that attempt.
• Payment API (auth · validation · idempotency): Public API: create payment, capture, refund, get status. Authenticates the merchant, validates the request, and enforces idempotency before any side effect happens. Stateless.
• Idempotency store (Redis (AOF) or Postgres table): Key: (merchant_id, idempotency_key). Value: request fingerprint, state (in_progress | done), and the final response. Prevents two concurrent or sequential requests with the same key from executing twice. Durable: losing this store reopens the double-charge hole.
• Payment service (state machine): Owns the payment lifecycle: created → authorized → captured → refunded / failed. Every transition is a transactional write to the Payments DB plus an outbox row. Calls the PSP through the adapter and never trusts a timeout as a definitive outcome.
• Payments DB (Postgres): Payment records, attempts, refunds, and the outbox table. Single-primary Postgres with synchronous replica for durability. Row-level locks on the payment row serialize concurrent transitions.
• Ledger service (double-entry): The source of truth for money. Records every movement as balanced debit/credit entries between accounts (buyer receivable, merchant payable, fee revenue, PSP clearing). Append-only. Balances are computed from entries, not stored as mutable columns.
• Ledger DB (Postgres · append-only): Journal and postings tables partitioned by month. Constraint: every journal entry's postings sum to zero. No UPDATE or DELETE grants for the application role. Periodic balance snapshots make "current balance" a cheap query.
• Card vault / tokenizer (PCI zone): The only place raw card numbers exist, in an isolated network with its own audit and access controls. Swaps a PAN for an opaque token; the rest of the system only ever sees tokens. Alternatively outsourced entirely to PSP-hosted fields.
• PSP adapter (Stripe / Adyen client): Wraps each provider behind one interface: authorize, capture, refund, void, get. Passes our idempotency key through to the PSP's own idempotency mechanism. Enforces timeouts, circuit breaking, and maps provider errors into our retryable / non-retryable / unknown classes.
• External PSP (card networks behind it): Stripe, Adyen, Braintree. Talks to card networks and issuing banks. Slow (hundreds of ms), occasionally times out, and sends asynchronous webhooks for events we did not observe synchronously (disputes, delayed captures, bank-side reversals).
• Event bus (Kafka · from outbox): Payment state-change events, published by an outbox relay so an event exists if and only if the DB transaction committed. Consumers: webhook dispatcher, ledger posting, analytics, fraud. Keyed by payment_id for per-payment ordering.
• Webhook dispatcher: Consumes events and delivers signed HTTP callbacks to merchant endpoints. Retries with exponential backoff for up to days, records every attempt, and exposes a replay API. At-least-once: merchants must dedupe on event id.
• Merchant backend: The merchant's own system. Receives webhooks to fulfil orders. Must verify the signature and dedupe on event id.
• Reconciliation job (nightly batch): Pulls the PSP settlement report, joins it against our payments and ledger by PSP transaction id, and produces three lists: matched, in-ours-not-theirs, in-theirs-not-ours, plus amount mismatches. Opens cases; never auto-corrects money.
• Payout service (scheduled): Computes each merchant's payable balance from the ledger (captured − refunds − fees − prior payouts), initiates a bank transfer via the PSP or a banking partner, and posts the payout to the ledger. Runs on a schedule with a reconciliation hold.

User flows

1. Card payment: authorize then capture. The main path. The buyer pays, we reserve funds at the PSP, record it durably, and later capture when the merchant ships. Every step is designed so that a crash anywhere leaves a recoverable state.
2. Retry after a timeout with the same idempotency key. The scenario that separates a real payment system from a toy: the client timed out, does not know whether the charge happened, and retries. There must be exactly one charge and one consistent answer.
3. Partial refund. Refunds are payments in reverse with an extra invariant: the sum of refunds can never exceed the captured amount, even under concurrent requests.
4. PSP webhook updates a payment asynchronously. The PSP tells us about things we did not observe synchronously: a delayed capture, a chargeback, a refund that finally settled. Webhooks arrive out of order, duplicated, and sometimes before our own transaction has committed.
5. Nightly reconciliation finds a mismatch. Even with idempotency and outboxes, bugs, PSP incidents and network partitions produce drift. Reconciliation is the safety net that turns unknown unknowns into tickets.

Deep dives

1. Idempotency key design. What exactly is stored under an idempotency key, for how long, and what happens on every kind of conflict?
2. Double-entry ledger vs a balance column. Why not just keep a balance column per merchant and add or subtract from it?
3. Payment state machine and the unknown outcome. The PSP call timed out. Did the charge happen? What do we do now?
4. Reliable events: the transactional outbox. The payment row committed but the Kafka publish failed. Now the ledger and the merchant never hear about it. How do we make "state changed" and "event published" atomic?
5. PCI scope and card tokenization. How do we accept card numbers without every server in the company falling under PCI DSS audit?
6. Consistency across PSP, payments DB and ledger. Three systems have to agree on every payment. Why not wrap them in a distributed transaction?