Design a Fraud Detection System

• Score every card payment inside the authorisation: One decision per attempt before the issuer is asked: allow, block, challenge with 3-D Secure, or allow and hold for review. Returned with a 0 to 99 score and reason codes inside the deadline the authorisation gives us.
• Real-time velocity and network features: Attempts, declines and amounts per card, device, IP, email and merchant over 1 minute, 1 hour and 24 hours; distinct cards per device; how this card has behaved at other merchants on the network.
• Merchant rules on top of the score: Merchants write allow, block, review and 3-D Secure rules over the score and named features, and see a backtest on their own history before a rule goes live.
• Platform rules and emergency blocks: The risk team can push a rule to every scorer within a minute during an attack, with an expiry so emergency rules do not outlive the attack.
• Manual review queue: Uncertain payments are authorised but not captured, ordered by expected loss and worked by analysts. A verdict captures or voids the payment and becomes a label.
• Labels from disputes and fraud reports: Chargebacks arrive 30 to 90 days later, issuer fraud reports within days or weeks, review verdicts within hours. Each is joined back to the exact decision it judges.
• Model lifecycle: Retrain on features as they were served, run new models in shadow, ramp gradually, roll back in a minute.
• Fraud ring detection: Offline graph analysis over cards, devices, emails and addresses. Ring membership becomes a feature at scoring time and context for analysts.
• Out of scope: The payment pipeline itself (idempotency keys, ledger, capture: see the Payment System), dispute evidence and representment, login and account takeover risk, sanctions and AML screening, merchant onboarding checks.

• Scoring latency (p99 < 100 ms): A hard deadline inside an authorisation that must finish in about 2 s. This drives the hardest trade-off in the design: every signal must be precomputed or computable in a few milliseconds, so anything expensive (graph analysis, long histories) moves offline and is read as a row.
• Throughput (10 k payments/s peak): About 300 M a day. Each payment reads ~15 feature keys and updates ~8 entities, so the feature store sees ~150 k reads and ~80 k writes a second at peak.
• Every payment gets a decision (100 % decided · ≥ 99.9 % by the full model): The risk system may degrade but never stops payments. A slow model or feature read falls back to rules only; failing closed for one minute at peak would decline about $30 M of payments.
• Feature freshness (velocity p99 < 2 s behind): Card-testing bots fire dozens of attempts a second, and a counter that lags by a minute sees none of them. The shortest attempt counters are also updated by the scorer itself.
• Point-in-time correct training (train on features as served): A training example must contain exactly what the scorer knew at decision time. Rebuilding features from today’s tables leaks chargebacks that arrived later into the past.
• Decision quality as a trade-off (fraud loss + false declines, per merchant): Zero fraud is easy: decline everything. The target is the lowest total of fraud losses plus the margin lost on good customers turned away, and the blocked side of that has to be measured, not assumed.
• No raw card numbers (tokens and keyed fingerprints only): The risk system stays outside PCI scope: the feature store, event log and training data hold a keyed hash of the card, the BIN and the last four digits, never the PAN. Merchants never see another merchant’s data.
• Every decision explainable (inputs + versions logged): Model version, rule versions, features as served and reason codes for each decision, so a merchant’s “why was this blocked?” and a regulator’s question have an exact answer.

• Payments per second: ~3.5 k avg · 10 k peak. 300 M payments/day ÷ 86 400 ≈ 3.5 k/s. The busiest hours and sale days run about 3× that: 10 k/s. Every one waits on a score, so the scoring tier is sized for peak with headroom, not for the average.
• Scoring budget by step: 100 ms p99 · ~15 ms typical. Caps inside the deadline: feature reads 30 ms, model 25 ms, rules 5 ms, request features and overhead 10 ms, network to and from the auth service 10 ms, slack 20 ms. Typical: features ~4 ms, model ~5 ms, rules ~1 ms. The caps exist for the tail, and each one has a fallback when it is hit.
• Feature reads: ~150 k keys/s at peak. ~15 keys per payment: ~12 entity rows (card, device, IP, email, customer, merchant, BIN, shipping address, and pairs such as card × merchant), the decision key and the 1-minute counters. 10 k/s × 15 = 150 k/s, ~20 KB per payment, ~200 MB/s. Sent as one batched multi-get, so it is one round trip per payment, not fifteen.
• Counter updates: ~80 k row writes/s at peak. Each attempt (allowed, blocked or declined by the issuer) updates ~8 entities: card, device, IP, email, customer, merchant, BIN, card × merchant. 10 k/s × 8 = 80 k/s row writes, each covering the minute, hour and day buckets at once. Hot keys (a big merchant, a carrier IP) are coalesced in the stream job and written ~10 times a second instead of 1,000.
• Online feature store: ~2 TB before replication. ~600 M entities active in the last 24 h × ~1 KB of window buckets ≈ 600 GB, plus ~3 B profile rows (cards, devices, emails, addresses seen in 18 months) × ~500 B ≈ 1.5 TB. About 2 TB, ~6 TB with 3 replicas per region: an SSD-backed key-value store, not RAM.
• Decision log for training: ~600 GB/day · ~330 TB kept. ~600 features as served plus score, rules fired and versions ≈ 2 KB per payment compressed. 300 M × 2 KB = 600 GB/day. Labels take 30 to 90 days (up to ~120), and training wants a year of matured payments plus last year’s season, so keep 18 months: 0.6 TB × ~550 days ≈ 330 TB in object storage, partitioned by day.
• Manual review volume: ~60 k cases/day. ~0.02 % of payments go to review: 300 M × 0.0002 = 60 k/day. At ~25 cases per analyst-hour that is ~2,400 analyst-hours, about 300 eight-hour shifts a day across merchants’ own teams and ours. Review is capacity-bound, so the queue is ordered by expected loss and the review line follows staffing.
• One minute failed closed: ~$30 M declined vs ~$50 k of fraud. At peak, 10 k/s × 60 s = 600 k payments × ~$50 = $30 M declined if risk fails closed. Failing open to rules only lets through the fraud the model catches beyond the rules: if ~0.4 % of value is attempted fraud and the model catches ~85 % against ~50 % for rules alone, that is 0.35 × 0.4 % × $30 M ≈ $42 k, about $50 k with dispute fees. Failing closed costs roughly 500× more, which settles the default.

• Card vault (PCI zone · HSM): The only place a raw card number exists. Turns the PAN into a token for the merchant and a card fingerprint: a keyed HMAC of the PAN with the key in an HSM, the same for the same card at every merchant. The fingerprint is what velocity counters, network features and the graph key on, which is how the risk system learns across merchants without ever being in PCI scope.
• Rules service (versioned · compiled · pushed): Stores platform and merchant rules (block, allow, review, 3-D Secure) over the score and named features, type-checks them against the feature registry, and pushes compiled, versioned rule sets to every scorer, so evaluating them is in memory and takes about a millisecond. Runs backtests on logged history before a rule goes live, and gives emergency rules an expiry.
• Model server (GBDT · live, shadow, previous): Serves a gradient-boosted tree ensemble over ~600 features in a few milliseconds on CPU, returning a calibrated fraud probability and the features that pushed it up, which become reason codes. Holds the live version, a shadow version scored after the decision is returned, and the previous version for a one-step rollback. Trees handle missing values natively, which is what makes the degraded-feature fallback safe.
• Training & backtests (weekly · label maturity): Retrains weekly on a year of logged decisions: payments older than ~90 days with their dispute labels, newer ones with early labels (issuer fraud reports, review verdicts) and their negatives down-weighted because some will still turn into chargebacks. Calibrates the output so a score means the same fraud rate in every version, evaluates on later weeks than it trains on, and replays rules over history for backtests.
• Decision log (object storage · by day): Every decision with its served features, versions and propensity, every outcome and every label, in columnar files partitioned by day and kept 18 months (~330 TB). The training set is a join of decisions to the labels known at a cutoff, by attempt id, so no feature is ever recomputed after the fact. Backtests and the graph job read it too.
• Merchant checkout (hosted card fields · device SDK): The merchant’s web or app checkout. The card is typed into the processor’s hosted fields, so the PAN goes straight to the vault and the merchant only ever holds a token. A device SDK collects browser and device signals and returns a device id that rides along with the payment, because the device is one of the strongest signals and fraudsters rotate devices far less often than cards.
• Auth service (authorisation path · ~2 s): The processor’s authorisation path (the Payment System question): idempotency, the payment state machine and the call to the card network. It asks the risk scorer for a decision with a 100 ms deadline before calling the issuer, then runs 3-D Secure, authorises or declines. If the scorer does not answer in time it applies a small static policy of its own, so risk can never take payments down.
• Risk scorer (stateless · owns the deadline): Runs one decision end to end: one batched read of ~15 feature keys, request-time features computed in process, the model call, platform and merchant rules over the score and features, then the expected-cost policy. It owns the 100 ms budget and every fallback (a missing key, a model timeout, a slow shard), and it logs exactly the feature vector it used, which is what training later learns from.
• Online feature store (SSD-backed KV · by entity): One row per entity (card fingerprint, device, IP, hashed email, merchant, pairs) holding window buckets written by the velocity jobs, profile and ring features loaded in batch, and a decision record per attempt for retries. Aerospike or ScyllaDB class: ~2 TB, single-digit millisecond p99. Rows hold time buckets rather than totals and the scorer sums the buckets inside each window when it reads, so a 1-hour count decays on its own with no event to trigger it.
• Graph job (daily + hourly delta): Builds a graph of cards, devices, emails, phones, shipping addresses and IPs linked by shared payments over 90 days, drops super-nodes (carrier IPs, freight forwarders), and computes connected components and proximity to confirmed fraud. Writes ring features (component size, share of disputed cards, hops to fraud) to the feature store, because a graph walk per payment would never fit in 100 ms.
• Card networks (issuers · 3-D Secure · disputes): Visa, Mastercard and the issuers behind them. They authorise or decline, run the 3-D Secure challenge with the cardholder, and weeks later send chargebacks with reason codes, plus fraud reports filed when a cardholder says a payment was not theirs. Slow, external, and the only source of the truth this system learns from.
• Review queue (priority by expected loss): Cases for payments authorised but held for a person, per merchant team and for the platform’s analysts, ordered by expected loss (probability × amount) and the time left before the authorisation expires. Shows linked entities from the graph, velocity history and reason codes. A verdict captures or voids the payment and is published as a label; cases nobody reaches in time fall back to the expected-cost decision.
• Velocity jobs (Flink · keyed by entity): Re-key each attempt into its ~8 entities and keep per-entity buckets of attempts, declines and amounts, plus distinct cards per device, IP and email (small exact sets that switch to a sparse HyperLogLog). Write absolute bucket values, not increments, deduplicated by attempt id, so a replay rewrites the same numbers. Coalesce hot keys to a few writes a second. p99 lag under 2 s.
• Labels service (disputes · fraud reports · verdicts): Normalises every signal that a payment was or was not fraud into label events per attempt: fraud-coded chargebacks, issuer fraud reports (Visa TC40, Mastercard SAFE), merchant refunds marked fraudulent, and review verdicts. Deduplicates by source id, keeps the time each label became known, and keeps non-fraud disputes (item not received) out of the fraud labels.
• Event log (Kafka · keyed by attempt id): Decisions with the features as served, attempt outcomes (issuer approved or declined, 3-D Secure passed or failed) and label events, keyed by attempt id so a decision and its outcome stay in order. The velocity jobs re-key by entity, a sink writes everything to the decision log, and the review service takes its cases from here, so nothing slow sits in the scoring path.

1. A card payment scored in 15 ms: The main path: the payment reaches the scorer with a fingerprint instead of a card number, one batched read fetches every feature, the model and rules decide, and the decision is logged with exactly what was used. Between the auth service and its answer there is one store read and one model call, nothing else remote.
   1. Checkout submits the payment with a card token and a device id; the auth service checks idempotency and starts the attempt.
   2. The auth service resolves the token to a card fingerprint, BIN and issuing country.
   3. The auth service asks the risk scorer for a decision with a 100 ms deadline.
   4. The scorer reads the decision key, ~12 entity rows and the 1-minute attempt counters in one batched call.
   5. The scorer computes request-time features and calls the model server.
   6. Platform and merchant rules run over the score and features, then the expected-cost policy picks the action.
   7. The scorer stores the decision under the attempt id, replies, and appends the decision with its exact features to the event log.
   8. The auth service authorises with the issuer and reports the outcome back to the scorer.
   9. The velocity jobs update the buckets of every entity in the attempt, and the sink appends it to the decision log.
2. A card-testing attack on a small merchant: A bot pushes 2,000 stolen card numbers through a small merchant at $1 each, about 30 a second, to learn which are live. It is the scale-breaking case for velocity features: thousands of attempts in a minute, each clean on its own, at a merchant that usually sees 20 payments a day.
   1. A bot starts sending 30 attempts a second of $1 payments with different card numbers through one merchant’s checkout.
   2. The 1-minute counters the scorer increments itself show the burst within the first second.
   3. Issuer declines flow back, and the velocity jobs update decline ratios and distinct cards per device within two seconds.
   4. Platform card-testing rules and the model start blocking: tiny amounts, a high decline ratio, many cards per device.
   5. The risk team pushes an emergency rule for this merchant, backtested on recent history in about a minute and set to expire in six hours.
   6. The cards the bot tested show up at other merchants within minutes, and network features mark them as compromised.
3. The model times out: rules-only fallback: A bad model deploy pushes inference past its 25 ms cap. Every payment still gets a decision within 100 ms; the question is which decision. This walks the fallback ladder from a slow feature shard to a scorer that does not answer at all, and the retry that must not be scored twice.
   1. One feature store shard slows down; the scorer hedges to a replica and marks keys that miss the 30 ms cap as missing.
   2. After a bad deploy the model server stops answering inside its 25 ms cap.
   3. The scorer decides with rules only: platform rules, merchant rules that do not use the score, and a conservative segment policy.
   4. Rules-only decisions are logged with their mode, and those payments are re-scored when the model is back, before capture.
   5. An authorisation’s risk call is retried after a timeout; the scorer finds its decision key and returns the same answer.
   6. If the scorer does not answer at all, the auth service applies its own static policy and goes on to the issuer.
4. Review, a fraud report, then a chargeback: The asynchronous loop that teaches the system: a held payment is reviewed within the hour, another payment is reported as fraud by its issuer after nine days and charged back after 47, and each signal is joined to the decision it judges. Then the graph job turns the confirmed fraud into features for the cards and devices linked to it.
   1. A $900 order is scored into the review band; it is authorised but not captured, and the case enters the merchant’s review queue.
   2. An analyst opens the case with its linked entities and velocity history, rejects it, and the review service voids the authorisation.
   3. The verdict becomes a label event, stamped with the time it became known.
   4. Nine days after a different payment, its issuer files a fraud report; on day 47 the chargeback arrives.
   5. The label updates the real-time features of that card, its devices and its email straight away.
   6. The sink appends the label to the decision log, where it joins the decision logged 47 days earlier by attempt id.
   7. The daily graph job links the disputed card to its devices, emails and addresses and writes ring features to the feature store.
5. Retrain, shadow and ramp a new model: A weekly retrain produces a model that is better offline. Getting it into production is the hard part, because the outcome it is judged on takes about 90 days to observe and blocked payments never get labels at all. Calibration, shadow scoring, a ramp by card and a long holdback replace the usual one-week A/B test.
   1. The training job builds a dataset from logged decisions and the labels known at the cutoff, with explored payments weighted by their propensity.
   2. It calibrates the new model so each score means the same fraud rate as before, and backtests it against the live model on held-out weeks.
   3. The new model is deployed to the model server as a shadow and scores every payment after the decision is returned.
   4. A week of shadow logs compares decisions, score distributions per merchant segment and latency.
   5. The scorer ramps the new model by card fingerprint hash, 1 % to 10 % to 50 % to 100 %, checking guardrails at each step.
   6. A 2 % holdback stays on the old model for 90 days, so the two models’ real chargeback rates can be compared when labels mature.

Ask: The model server or a feature shard is slow. Do you wait, decline the payment, or let it through, and what does each choice cost?

Good answers name: Per-step caps with a fallback ladder: missing features, then rules only, then the auth service’s static policy, Fail closed: decline or challenge anything the model did not score, Wait for the model: extend the deadline when it is slow, Authorise first and score asynchronously before capture.

Our pick: The auth service propagates a 100 ms deadline and the scorer splits it into caps: 30 ms for the batched feature read (hedged at the shard’s p95), 25 ms for the model, 5 ms for rules, the rest for overhead, network and slack. A key that misses its cap becomes a missing value, which the tree model handles. A model that misses its cap means rules only: blocklists, velocity and card-testing rules, merchant rules that do not use the score, and a segment policy that challenges or declines high-risk segments and allows the rest. A circuit breaker stops calling a model that keeps failing. If the scorer does not answer, the auth service applies a static policy of its own. The rules-only decision is computed and logged for every payment, always, so its quality is known and the path is never cold. Payments decided in a degraded mode are re-scored when the model returns and held for review before capture if they cross the line. The SLO is two numbers: 100 % of payments decided, and at least 99.9 % decided by the full model.

1. Can an attacker force the fallback on purpose?
   Only if something in their request controls latency, so nothing does: the feature read is a fixed set of keys, input sizes are capped, and model cost does not depend on values. An attacker cannot make their own payment slow. They can wait for an incident, which is why rules only is not "allow everything": card-testing rules, blocklists and segment challenges still apply, and degraded payments are re-scored before capture.
2. Why not run the model inside the scorer process and remove a network hop?
   That is a reasonable design and saves a millisecond or two and one failure mode. I kept a separate model server so shadow models run on their own capacity without slowing the live path, so a model rollout is not a scorer deploy, and so a model that blows up its memory takes down the model tier rather than the process that holds the fallback. If models were tiny and rarely changed, in process would win.
3. A whole region’s risk tier is down for ten minutes. What happens?
   The auth services in that region fall back to their static policy within one timeout, and traffic can be steered to the risk tier in another region if the extra round trip still fits the deadline. Ten minutes of static policy at peak is on the order of half a million dollars of extra fraud exposure, much of which the re-score before capture claws back. Every unscored attempt is logged, so the loss is measured, not guessed.
4. How do you know the fallback policy is any good?
   Because it runs on every payment in shadow. For each payment we log the decision the rules-only path would have made, and when labels mature we know its fraud catch rate and false declines exactly, per segment. That number is what goes into the cost of an incident, and when it drifts we tune the segment policy instead of discovering the problem during an outage.
5. At 10× the traffic, what breaks first?
   The feature read, not the model. 1.5 M key reads a second is fine for a sharded store, but tail latency grows with fan-out, so I would keep each payment’s keys on as few shards as possible (pair keys stored with their card) and keep hedging. The model tier scales horizontally on CPU, and the velocity jobs scale with partitions; the hot keys are already coalesced.

Ask: A device has used 40 cards in the last hour. How does the scorer know that within a second, for a billion devices, cards and IPs?

Good answers name: Stream jobs keep per-entity time buckets in a key-value store; the scorer sums the buckets inside each window at read time, Increment the 1-minute attempt counters from the scoring path, Query the payments database per request, Exact sliding windows from a list of event timestamps per entity.

Our pick: A Flink job re-keys each attempt into its ~8 entities. Each entity row holds bucket arrays: six 10-second buckets for the minute, twelve 5-minute buckets for the hour and 24 hourly buckets for the day, for attempts, declines and amount, each stamped with its start time. The job writes absolute bucket values, deduplicated by attempt id in its state, and the scorer sums the buckets that fall inside each window as it reads, so windows decay with no event. Distinct counts (cards per device, IP and email; merchants per card) are small exact sets of hashed ids up to 32 entries, then a sparse HyperLogLog, because almost every device has one or two cards and the ones past 32 are exactly the interesting ones. On top, the scorer increments the 1-minute attempt buckets for the card, device, IP and merchant right after its read, without waiting, and only when the decision key shows this is not a retry. Hot keys are pre-aggregated per 100 ms in the job. Everything is keyed by fingerprint, device id or hashed email, never a card number. Stripe has written publicly that Radar learns from payments across its whole network, which is the same idea as these cross-merchant features; its internals are not public, so the specifics here are my own design.

1. The stream job falls five minutes behind. What happens to decisions?
   Every feature row carries its last-write time, and the scorer logs feature age with each decision. Past a freshness limit (30 s for the 1-minute windows) a window is treated as missing rather than trusted, because a stale zero looks exactly like a quiet card. The scorer’s own 1-minute increments keep working, so card testing is still caught. Consumer lag and p99 feature age are paged alerts, and once the job catches up the buckets are right again because its writes are absolute.
2. A mobile carrier puts 50,000 customers behind one IP. Doesn’t IP velocity block all of them?
   Raw counts on a carrier NAT or a corporate proxy are meaningless, so IP features are relative: this IP’s attempts against its own baseline for that hour of the day, plus ASN and known-proxy flags. The graph job and an IP reputation table mark shared IPs, and the model learns that a high count on a carrier IP means little. Device and card features carry the signal there instead.
3. How big is "distinct cards per device" for a billion devices?
   Almost all devices have one to three cards, so a small exact set of 4-byte hashed ids per hourly bucket costs a few dozen bytes. Only devices past 32 distinct cards switch to a sparse HyperLogLog of a few hundred bytes. A full 12 KB HyperLogLog per device would be 12 TB for a billion devices; the hybrid stays inside the ~600 GB of window state.
4. Why not use the stream processor’s own sliding windows and emit totals?
   A sliding window that emits on every step has to emit for every key every step to stay current, which is billions of writes for keys nobody is reading, or it goes stale when a key goes quiet. Buckets written only when an event arrives, and summed at read time, give the same answer with writes proportional to traffic. It is the usual trick of building sliding windows from tumbling ones, moved to the read side.
5. A card has no row at all. Is that a zero?
   No, and the model must be able to tell. "Never seen on the network" is one of the strongest features there is, because most good cards have history somewhere. A missing row is encoded as never seen, while a row of zeros is a known card having a quiet hour. Collapsing the two is a classic way to make a model blind to new cards.

Ask: A chargeback arrives 47 days after the payment. How do you turn it into a training example, and what goes wrong if you rebuild the features from today’s tables?

Good answers name: Train on logged served features joined to labels by attempt id, with label maturity and early-label proxies, Recompute features from the warehouse with point-in-time joins, Train on all payments with the labels known today, Train only on fully matured payments, older than 120 days.

Our pick: The scorer logs every decision with its feature vector exactly as served (values, missing mask, feature age), the model and rule versions, the mode and the propensity of the action taken, keyed by attempt id. The labels service emits label events with a source, a reason code and known_at. A weekly training job takes about a year of decisions: payments older than ~90 days use dispute labels; payments between 7 and 90 days old use issuer fraud reports, merchant fraud refunds and review verdicts, with their negatives down-weighted by the share of fraud that historically has not been reported yet at that age. Evaluation is out of time: train on earlier weeks and test on later ones with only the labels known at each test date, because a random split puts the same fraud ring on both sides. New features are backfilled by running the velocity job’s code over archived events until a month of served values exists. The cost is storage, ~330 TB, which is cheap next to a model that is wrong in a way no offline metric shows.

1. A fraud report arrives but the chargeback never does. Is it fraud?
   Usually yes. An issuer fraud report means the cardholder told their bank the payment was not theirs, and many are never disputed because the amount is small or the issuer absorbs it. I treat it as a positive with slightly lower weight than a fraud chargeback. A non-fraud dispute such as "item not received" is not a fraud label at all, which is why the labels service keeps reason codes.
2. What about friendly fraud, a real cardholder disputing a payment they made?
   It arrives with a fraud reason code and looks identical in the labels. The tell is that the card, device and shipping address have a long good history together and the customer kept buying afterwards. I would keep it as a separate label class rather than training on it as stolen-card fraud, because blocking a loyal customer’s next payment is the wrong answer to a refund dispute.
3. Why weekly retraining and not daily?
   The labels that matter most mature over months, so a daily retrain mostly reshuffles the same data, and each new model needs a shadow week and a ramp anyway. Fast-moving attacks are handled by rules and by real-time features such as card-testing counters, which react in seconds, not by retraining. If the score drifts between weekly models, recalibrating on early labels is the cheaper fix.
4. Why an out-of-time evaluation instead of a random split?
   Fraud comes in campaigns: the same ring, devices and BIN range across many payments in a few days. A random split puts half a campaign in training and half in test, and the model looks clever for recognising it. An out-of-time split asks the real question: does a model trained on the past catch next month’s fraud?

Ask: Your model blocks a payment, so you never learn whether it was fraud. How do you measure your false declines, and how do you stop the next model forgetting what this one blocks?

Good answers name: A small randomised exploration budget in the borderline band, with propensities logged, Challenge with 3-D Secure instead of blocking in the borderline band, Train only on allowed payments, Treat every blocked payment as fraud.

Our pick: In the band just above each merchant’s block line (within ~10 score points, amount under $100, never for blocklists or hard platform rules, never for merchants who opt out), 1 in 200 would-be blocks is allowed instead. That is ~1,500 payments a day; at a ~25 % fraud rate and ~$55 per fraud including fees, about $20 k a day or ~$7.5 M a year, which the platform reimburses rather than the merchant. Whether a payment is explored is decided by a hash of the card fingerprint and the day, so retrying the same card does not reroll it. Every decision logs the probability of the action it took (1.0 for ordinary decisions, 0.005 for an explored allow), so training weights explored examples by 1 / propensity and a new policy can be evaluated offline with inverse propensity scoring over logged decisions. Where 3-D Secure is available, challenges replace blocks in the lower band and give cheaper labels; reviewer verdicts help too, but only inside the review band. The measured fraud rate in the explored band is what sets the block line, so a few million dollars a year buys the ability to tune a decision worth millions a day.

1. A merchant asks why you let a fraudulent payment through on purpose.
   Because without it we cannot tell them how many good customers we are turning away, and that number is usually larger than their fraud. The budget is tiny, capped by amount, never applied to hard blocks, and we refund any loss from an explored payment, so the merchant is never worse off. Merchants who still object can opt out; their thresholds are then set from the rest of the network’s exploration.
2. Couldn’t a fraudster exploit exploration?
   Only by getting lucky, because exploration is decided per card per day, not per attempt: a card that is not explored today stays blocked on every retry. Amounts are capped, hard rules never explore, and a card that turns out fraudulent feeds back into its features within seconds. At 1 in 200, a fraudster burns 200 cards to get one $100 payment through.
3. How do you evaluate a new threshold offline before using it?
   Replay it over logged decisions. Where the old and new policy agree, the observed outcome counts as it is. Where they disagree, only explored payments say what would have happened, weighted by 1 / propensity. The variance is high for big policy changes, which is a reason to move thresholds in small steps and confirm with a holdback.
4. Why not explore at high scores too?
   The fraud rate there is so high that each explored payment buys little information at a high price, and the decision is not close: we would block at 80 % under any reasonable cost. A label is worth most near the line, where a few points of calibration error move hundreds of thousands of decisions a day, so that is where the budget goes.

Ask: Why not one global threshold on the score, and why let merchants write their own rules on top of your model?

Good answers name: Calibrated probability, expected-cost thresholds per merchant, and rules over score and features, backtested and shadowed, One global threshold on the score, Rules only, no model, A separate model per merchant.

Our pick: The model returns a calibrated probability p, exposed to merchants as a 0 to 99 score on a fixed scale. The default policy is expected cost: allowing costs p × (amount + ~$15 fee); declining costs (1 − p) × margin × amount, with the margin from the merchant’s settings or their category. So decline when p > mA / (mA + A + F). For a $100 payment at a 30 % margin that is about 21 %; for the $800 laptop at 5 % it is under 5 %; for the $20 licence at 80 % it is about 31 %. Where 3-D Secure is available a challenge sits in between, because it costs about 10 % of the good customers who get one and moves liability for most fraud to the issuer: for the $100 example, challenge from about 2.5 % and decline only far higher, where even a challenged payment loses money. Rules run on top in a fixed order: platform hard rules, then the merchant’s allow list, then merchant block, review and challenge rules, then the default policy. A rule is type-checked against the feature registry, backtested on 90 days of the merchant’s served features (hits, fraud among them, good payments among them, and hits whose outcome is unknown because something else blocked them), run in log-only mode for a few days, then enforced.

1. A merchant writes “block if card country ≠ IP country”. What does the backtest show them?
   That it hits perhaps 8 % of their payments, mostly travellers, expats and people on corporate VPNs, of which a small share were fraud, so it would turn away many good customers for every fraud it stops, most of which the model already catches. We show the hits, the fraud and good payments among them, and the margin it would cost next to what it adds. Most merchants then rewrite it as a review or challenge rule with a score condition.
2. You ship a new model. Do every merchant’s “score > 65” rules change meaning?
   Not if the score is calibrated to a fixed scale: 65 is pinned to the same fraud probability in every version. The training job fits that mapping on recent matured data, and the backtest checks per merchant segment that rule hit rates move only where the new model genuinely ranks payments differently. Without calibration, a model upgrade silently rewrites thousands of merchants’ rules.
3. Can a merchant allow list override a platform block?
   No. Platform hard rules (cards confirmed stolen, active card-testing guards) run first and cannot be overridden, because their cost does not land only on that merchant: it lands on issuers and the network, and the merchant that allows it becomes the hole every fraudster uses. A merchant allow list overrides the model and the default policy for their own trusted customers, which is what it is for.
4. How fast must a rule change take effect?
   Seconds for emergency rules during an attack, minutes for ordinary ones. Rule sets are versioned and pushed to every scorer as deltas, each decision logs the version it used, and emergency rules carry an expiry so they do not outlive the attack. A bad rule is rolled back by version, like a deploy.

Ask: A ring uses hundreds of fresh cards, each clean on its own. How do you catch the first payment from a card you have never seen?

Good answers name: Offline graph job: components and proximity to fraud as per-entity features, daily plus an hourly delta, A graph database queried per payment, Graph neural network embeddings per entity, Per-entity velocity only, no graph.

Our pick: Every attempt in the decision log yields links between its entities: card to device, card to email, device to IP, card to shipping address. A daily Spark job builds the 90-day graph, drops super-nodes (any IP, address or device fingerprint linked to more than ~1,000 cards, plus known carrier, proxy and freight-forwarder lists), weights links by recency, and computes connected components and a personalised PageRank seeded from confirmed fraud. Each entity gets features: component size, the number and share of disputed cards in its component, hops to the nearest confirmed fraud, and the age of its first link. These are bulk-loaded into the feature store under a version, and an hourly delta propagates new links to entities within one hop of confirmed fraud. Analysts see the same component in the review tool. The guard against false links is that these are features, not rules: the model learns how much a shared device means relative to everything else.

1. A family shares one laptop and three cards. Do they look like a ring?
   They form a small component with no disputed cards, which is the most common shape on the network, and the model learns it is safe. Rings look different: many cards, few devices, several drop addresses, and disputes accumulating inside the component. The share of disputed cards and the hops to fraud matter far more than component size on its own.
2. One shared resource merges two big components. What then?
   That is the super-node problem in a smaller form: a co-working space’s IP or a reused gift-card address. Pruning by degree catches the obvious ones; for the rest, links are weighted by how rare the shared resource is, so a component held together by one common link splits. The largest component’s size is monitored: if it suddenly grows by millions, a new super-node slipped through.
3. Why not run the graph job in real time?
   The parts that need to be real time already are: a confirmed fraud updates its card, device and email features within seconds through the velocity jobs, and the hourly delta handles one-hop neighbours. Recomputing components over tens of billions of edges continuously is expensive and buys little, because rings are built over days and weeks.
4. Does linking cards across merchants raise privacy problems?
   Yes, so the graph holds only keyed fingerprints and hashed identifiers, never card numbers, emails or addresses in the clear, and merchants see a score and reasons, never another merchant’s customers. Analysts see linked entities as masked ids with risk facts (disputed, blocked), not other merchants’ order details. Data is kept only for the window the model needs and deleted on request where the law requires.