Design a Fraud Detection System

Allow, block or challenge 10,000 card payments a second in under 100 ms, then learn from chargebacks that arrive months later.

Last updated 2026-10-27. Difficulty: hard. Patterns: fraud, feature-store, stream-processing, low-latency, exploration. Reported at Stripe, PayPal, Coinbase, Amazon, Uber, Airbnb.

Functional requirements

• Score every card payment inside the authorisation. One decision per attempt before the issuer is asked: allow, block, challenge with 3-D Secure, or allow and hold for review. Returned with a 0 to 99 score and reason codes inside the deadline the authorisation gives us.
• Real-time velocity and network features. Attempts, declines and amounts per card, device, IP, email and merchant over 1 minute, 1 hour and 24 hours; distinct cards per device; how this card has behaved at other merchants on the network.
• Merchant rules on top of the score. Merchants write allow, block, review and 3-D Secure rules over the score and named features, and see a backtest on their own history before a rule goes live.
• Platform rules and emergency blocks. The risk team can push a rule to every scorer within a minute during an attack, with an expiry so emergency rules do not outlive the attack.
• Manual review queue. Uncertain payments are authorised but not captured, ordered by expected loss and worked by analysts. A verdict captures or voids the payment and becomes a label.
• Labels from disputes and fraud reports. Chargebacks arrive 30 to 90 days later, issuer fraud reports within days or weeks, review verdicts within hours. Each is joined back to the exact decision it judges.
• Model lifecycle. Retrain on features as they were served, run new models in shadow, ramp gradually, roll back in a minute.
• Fraud ring detection. Offline graph analysis over cards, devices, emails and addresses. Ring membership becomes a feature at scoring time and context for analysts.
• Out of scope. The payment pipeline itself (idempotency keys, ledger, capture: see the Payment System), dispute evidence and representment, login and account takeover risk, sanctions and AML screening, merchant onboarding checks.

Non-functional requirements

• Scoring latency (p99 < 100 ms). A hard deadline inside an authorisation that must finish in about 2 s. This drives the hardest trade-off in the design: every signal must be precomputed or computable in a few milliseconds, so anything expensive (graph analysis, long histories) moves offline and is read as a row.
• Throughput (10 k payments/s peak). About 300 M a day. Each payment reads ~15 feature keys and updates ~8 entities, so the feature store sees ~150 k reads and ~80 k writes a second at peak.
• Every payment gets a decision (100 % decided · ≥ 99.9 % by the full model). The risk system may degrade but never stops payments. A slow model or feature read falls back to rules only; failing closed for one minute at peak would decline about $30 M of payments.
• Feature freshness (velocity p99 < 2 s behind). Card-testing bots fire dozens of attempts a second, and a counter that lags by a minute sees none of them. The shortest attempt counters are also updated by the scorer itself.
• Point-in-time correct training (train on features as served). A training example must contain exactly what the scorer knew at decision time. Rebuilding features from today’s tables leaks chargebacks that arrived later into the past.
• Decision quality as a trade-off (fraud loss + false declines, per merchant). Zero fraud is easy: decline everything. The target is the lowest total of fraud losses plus the margin lost on good customers turned away, and the blocked side of that has to be measured, not assumed.
• No raw card numbers (tokens and keyed fingerprints only). The risk system stays outside PCI scope: the feature store, event log and training data hold a keyed hash of the card, the BIN and the last four digits, never the PAN. Merchants never see another merchant’s data.
• Every decision explainable (inputs + versions logged). Model version, rule versions, features as served and reason codes for each decision, so a merchant’s “why was this blocked?” and a regulator’s question have an exact answer.

Back-of-envelope estimates

• Payments per second: ~3.5 k avg · 10 k peak. 300 M payments/day ÷ 86 400 ≈ 3.5 k/s. The busiest hours and sale days run about 3× that: 10 k/s. Every one waits on a score, so the scoring tier is sized for peak with headroom, not for the average.
• Scoring budget by step: 100 ms p99 · ~15 ms typical. Caps inside the deadline: feature reads 30 ms, model 25 ms, rules 5 ms, request features and overhead 10 ms, network to and from the auth service 10 ms, slack 20 ms. Typical: features ~4 ms, model ~5 ms, rules ~1 ms. The caps exist for the tail, and each one has a fallback when it is hit.
• Feature reads: ~150 k keys/s at peak. ~15 keys per payment: ~12 entity rows (card, device, IP, email, customer, merchant, BIN, shipping address, and pairs such as card × merchant), the decision key and the 1-minute counters. 10 k/s × 15 = 150 k/s, ~20 KB per payment, ~200 MB/s. Sent as one batched multi-get, so it is one round trip per payment, not fifteen.
• Counter updates: ~80 k row writes/s at peak. Each attempt (allowed, blocked or declined by the issuer) updates ~8 entities: card, device, IP, email, customer, merchant, BIN, card × merchant. 10 k/s × 8 = 80 k/s row writes, each covering the minute, hour and day buckets at once. Hot keys (a big merchant, a carrier IP) are coalesced in the stream job and written ~10 times a second instead of 1,000.
• Online feature store: ~2 TB before replication. ~600 M entities active in the last 24 h × ~1 KB of window buckets ≈ 600 GB, plus ~3 B profile rows (cards, devices, emails, addresses seen in 18 months) × ~500 B ≈ 1.5 TB. About 2 TB, ~6 TB with 3 replicas per region: an SSD-backed key-value store, not RAM.
• Decision log for training: ~600 GB/day · ~330 TB kept. ~600 features as served plus score, rules fired and versions ≈ 2 KB per payment compressed. 300 M × 2 KB = 600 GB/day. Labels take 30 to 90 days (up to ~120), and training wants a year of matured payments plus last year’s season, so keep 18 months: 0.6 TB × ~550 days ≈ 330 TB in object storage, partitioned by day.
• Manual review volume: ~60 k cases/day. ~0.02 % of payments go to review: 300 M × 0.0002 = 60 k/day. At ~25 cases per analyst-hour that is ~2,400 analyst-hours, about 300 eight-hour shifts a day across merchants’ own teams and ours. Review is capacity-bound, so the queue is ordered by expected loss and the review line follows staffing.
• One minute failed closed: ~$30 M declined vs ~$50 k of fraud. At peak, 10 k/s × 60 s = 600 k payments × ~$50 = $30 M declined if risk fails closed. Failing open to rules only lets through the fraud the model catches beyond the rules: if ~0.4 % of value is attempted fraud and the model catches ~85 % against ~50 % for rules alone, that is 0.35 × 0.4 % × $30 M ≈ $42 k, about $50 k with dispute fees. Failing closed costs roughly 500× more, which settles the default.

Components

• Card vault (PCI zone · HSM): The only place a raw card number exists. Turns the PAN into a token for the merchant and a card fingerprint: a keyed HMAC of the PAN with the key in an HSM, the same for the same card at every merchant. The fingerprint is what velocity counters, network features and the graph key on, which is how the risk system learns across merchants without ever being in PCI scope.
• Rules service (versioned · compiled · pushed): Stores platform and merchant rules (block, allow, review, 3-D Secure) over the score and named features, type-checks them against the feature registry, and pushes compiled, versioned rule sets to every scorer, so evaluating them is in memory and takes about a millisecond. Runs backtests on logged history before a rule goes live, and gives emergency rules an expiry.
• Model server (GBDT · live, shadow, previous): Serves a gradient-boosted tree ensemble over ~600 features in a few milliseconds on CPU, returning a calibrated fraud probability and the features that pushed it up, which become reason codes. Holds the live version, a shadow version scored after the decision is returned, and the previous version for a one-step rollback. Trees handle missing values natively, which is what makes the degraded-feature fallback safe.
• Training & backtests (weekly · label maturity): Retrains weekly on a year of logged decisions: payments older than ~90 days with their dispute labels, newer ones with early labels (issuer fraud reports, review verdicts) and their negatives down-weighted because some will still turn into chargebacks. Calibrates the output so a score means the same fraud rate in every version, evaluates on later weeks than it trains on, and replays rules over history for backtests.
• Decision log (object storage · by day): Every decision with its served features, versions and propensity, every outcome and every label, in columnar files partitioned by day and kept 18 months (~330 TB). The training set is a join of decisions to the labels known at a cutoff, by attempt id, so no feature is ever recomputed after the fact. Backtests and the graph job read it too.
• Merchant checkout (hosted card fields · device SDK): The merchant’s web or app checkout. The card is typed into the processor’s hosted fields, so the PAN goes straight to the vault and the merchant only ever holds a token. A device SDK collects browser and device signals and returns a device id that rides along with the payment, because the device is one of the strongest signals and fraudsters rotate devices far less often than cards.
• Auth service (authorisation path · ~2 s): The processor’s authorisation path (the Payment System question): idempotency, the payment state machine and the call to the card network. It asks the risk scorer for a decision with a 100 ms deadline before calling the issuer, then runs 3-D Secure, authorises or declines. If the scorer does not answer in time it applies a small static policy of its own, so risk can never take payments down.
• Risk scorer (stateless · owns the deadline): Runs one decision end to end: one batched read of ~15 feature keys, request-time features computed in process, the model call, platform and merchant rules over the score and features, then the expected-cost policy. It owns the 100 ms budget and every fallback (a missing key, a model timeout, a slow shard), and it logs exactly the feature vector it used, which is what training later learns from.
• Online feature store (SSD-backed KV · by entity): One row per entity (card fingerprint, device, IP, hashed email, merchant, pairs) holding window buckets written by the velocity jobs, profile and ring features loaded in batch, and a decision record per attempt for retries. Aerospike or ScyllaDB class: ~2 TB, single-digit millisecond p99. Rows hold time buckets rather than totals and the scorer sums the buckets inside each window when it reads, so a 1-hour count decays on its own with no event to trigger it.
• Graph job (daily + hourly delta): Builds a graph of cards, devices, emails, phones, shipping addresses and IPs linked by shared payments over 90 days, drops super-nodes (carrier IPs, freight forwarders), and computes connected components and proximity to confirmed fraud. Writes ring features (component size, share of disputed cards, hops to fraud) to the feature store, because a graph walk per payment would never fit in 100 ms.
• Card networks (issuers · 3-D Secure · disputes): Visa, Mastercard and the issuers behind them. They authorise or decline, run the 3-D Secure challenge with the cardholder, and weeks later send chargebacks with reason codes, plus fraud reports filed when a cardholder says a payment was not theirs. Slow, external, and the only source of the truth this system learns from.
• Review queue (priority by expected loss): Cases for payments authorised but held for a person, per merchant team and for the platform’s analysts, ordered by expected loss (probability × amount) and the time left before the authorisation expires. Shows linked entities from the graph, velocity history and reason codes. A verdict captures or voids the payment and is published as a label; cases nobody reaches in time fall back to the expected-cost decision.
• Velocity jobs (Flink · keyed by entity): Re-key each attempt into its ~8 entities and keep per-entity buckets of attempts, declines and amounts, plus distinct cards per device, IP and email (small exact sets that switch to a sparse HyperLogLog). Write absolute bucket values, not increments, deduplicated by attempt id, so a replay rewrites the same numbers. Coalesce hot keys to a few writes a second. p99 lag under 2 s.
• Labels service (disputes · fraud reports · verdicts): Normalises every signal that a payment was or was not fraud into label events per attempt: fraud-coded chargebacks, issuer fraud reports (Visa TC40, Mastercard SAFE), merchant refunds marked fraudulent, and review verdicts. Deduplicates by source id, keeps the time each label became known, and keeps non-fraud disputes (item not received) out of the fraud labels.
• Event log (Kafka · keyed by attempt id): Decisions with the features as served, attempt outcomes (issuer approved or declined, 3-D Secure passed or failed) and label events, keyed by attempt id so a decision and its outcome stay in order. The velocity jobs re-key by entity, a sink writes everything to the decision log, and the review service takes its cases from here, so nothing slow sits in the scoring path.

User flows

1. A card payment scored in 15 ms. The main path: the payment reaches the scorer with a fingerprint instead of a card number, one batched read fetches every feature, the model and rules decide, and the decision is logged with exactly what was used. Between the auth service and its answer there is one store read and one model call, nothing else remote.
   1. Checkout submits the payment with a card token and a device id; the auth service checks idempotency and starts the attempt. The card was entered in hosted fields, so the merchant and the auth service hold only a token. The device id comes from the device SDK and is worth as much as the card: fraudsters buy cards by the thousand but rotate devices far less. The attempt id minted here is the key every later step uses.
   2. The auth service resolves the token to a card fingerprint, BIN and issuing country. The fingerprint is a keyed HMAC of the PAN computed inside the vault, the same for the same card at every merchant, so counters and network features can follow a card across the whole processor without the risk system ever holding the number. A plain hash would not do: with the BIN and last four known, only about 100,000 card numbers remain to try.
   3. The auth service asks the risk scorer for a decision with a 100 ms deadline. The request carries everything known at this moment: amount, merchant, fingerprint, BIN, device, IP, hashed email and shipping address. The deadline is propagated, so the scorer knows how much time it really has left after the hop. The authorisation as a whole has about 2 s, almost all of it reserved for the issuer and 3-D Secure.
   4. The scorer reads the decision key, ~12 entity rows and the 1-minute attempt counters in one batched call. One multi-get, fanned out to the ~4 shards that hold the keys and hedged to a replica at the shard’s p95, so it takes ~4 ms typically and is capped at 30 ms. Rows hold buckets (six 10 s buckets for the minute, twelve 5 min buckets for the hour, 24 hourly buckets for the day) and the scorer sums those inside each window. Right after the read, and only if the decision key was absent, it increments the 1-minute attempt buckets for the card, device, IP and merchant without waiting, so the next attempt in a burst sees this one before the stream has caught up.
   5. The scorer computes request-time features and calls the model server. Request-time features come from tables held in memory: IP to country and ASN, BIN to issuer and card type, distance between IP and billing country, age of the email domain. Then ~600 features go to a gradient-boosted ensemble that returns a calibrated probability in ~5 ms. The call is capped at 25 ms; past that the scorer decides with rules only.
   6. Platform and merchant rules run over the score and features, then the expected-cost policy picks the action. Rules are compiled and held in memory, so this takes about a millisecond. Order: platform hard rules (confirmed stolen cards, card-testing guards), then the merchant’s allow list, then the merchant’s block, review and challenge rules, then the default policy, which compares p × (amount + dispute fee) with the margin lost by declining a good customer. Here p = 1.8 % on a $129 payment is below this merchant’s challenge line, so the answer is allow.
   7. The scorer stores the decision under the attempt id, replies, and appends the decision with its exact features to the event log. A set-if-absent on the decision key with a 24 h TTL comes before the reply, so a retried call gets the same answer. The log write is asynchronous and off the response path, but the event holds the feature vector as served, the model and rule versions, the mode (full, degraded, rules only) and the probability of the action taken, which training and exploration need later.
   8. The auth service authorises with the issuer and reports the outcome back to the scorer. The issuer’s answer is part of the signal: a run of "do not honour" and "invalid card" declines at one merchant is the signature of card testing. The outcome is reported fire-and-forget and lands on the event log under the same attempt id, right behind its decision.
   9. The velocity jobs update the buckets of every entity in the attempt, and the sink appends it to the decision log. The job re-keys the attempt into its ~8 entities and rewrites each row’s current buckets as absolute values, deduplicated by attempt id, within 1 to 2 s of the outcome. The decision log gets the decision, the outcome and, months later, the label, all joinable by attempt id.
2. A card-testing attack on a small merchant. A bot pushes 2,000 stolen card numbers through a small merchant at $1 each, about 30 a second, to learn which are live. It is the scale-breaking case for velocity features: thousands of attempts in a minute, each clean on its own, at a merchant that usually sees 20 payments a day.
   1. A bot starts sending 30 attempts a second of $1 payments with different card numbers through one merchant’s checkout. Each card is new to this merchant and many are new to the network, the amounts are tiny, and the IPs rotate through a proxy pool. Per payment, nothing looks extreme; the signal is the shape of the traffic. The merchant normally sees 20 payments a day, so 30 a second is a jump of over 100,000×.
   2. The 1-minute counters the scorer increments itself show the burst within the first second. The stream lags by 1 to 2 s, which at 30 attempts a second is up to 60 attempts nobody has counted yet. The scorer’s own increments on the merchant’s and device’s 10-second buckets land within milliseconds, so attempts per merchant per minute and attempts per device are visible from the first handful of payments.
   3. Issuer declines flow back, and the velocity jobs update decline ratios and distinct cards per device within two seconds. Most stolen numbers are dead, so 70 to 80 % of attempts are declined by issuers: a merchant decline ratio over one minute that is normally under 10 % jumps to 75 %. The merchant row is now hot, so the job pre-aggregates its updates per 100 ms and writes it ten times a second, with exact counts held in its state.
   4. Platform card-testing rules and the model start blocking: tiny amounts, a high decline ratio, many cards per device. A platform rule such as "block if merchant decline ratio over 1 minute > 50 % and amount < $5 and card first seen on the network under 24 h ago" fires on top of a model score that is already high. Blocking here is nearly free: a false decline on a $1 payment at a merchant under attack costs almost nothing. The bot now gets blocks instead of issuer answers, so it learns nothing about which cards are live.
   5. The risk team pushes an emergency rule for this merchant, backtested on recent history in about a minute and set to expire in six hours. Even mid-attack, the backtest answers the question that matters: how many of the merchant’s normal payments would this have blocked? Here it hits 1,950 of the 2,000 attempts since the attack began and 0.2 % of the merchant’s previous 30 days. The compiled rule reaches every scorer within seconds, and the expiry keeps emergency rules from piling up for years.
   6. The cards the bot tested show up at other merchants within minutes, and network features mark them as compromised. Testers sell or use the cards that passed. Every attempt updated the card fingerprint’s row, so a card tried at a merchant under attack an hour ago now carries that fact everywhere: tested at a flagged merchant, declined 3 times in an hour across 2 merchants. One merchant’s attack protects every other merchant on the network.
3. The model times out: rules-only fallback. A bad model deploy pushes inference past its 25 ms cap. Every payment still gets a decision within 100 ms; the question is which decision. This walks the fallback ladder from a slow feature shard to a scorer that does not answer at all, and the retry that must not be scored twice.
   1. One feature store shard slows down; the scorer hedges to a replica and marks keys that miss the 30 ms cap as missing. A hedged read at the shard’s p95 recovers most of the tail. Keys still missing at 30 ms go to the model as missing values, which gradient-boosted trees handle with a learned default branch, and the decision is logged as mode "degraded" with the missing keys named, so the cost of the incident can be measured later.
   2. After a bad deploy the model server stops answering inside its 25 ms cap. Without a cap, a slow model would eat the whole 100 ms and the auth service’s own timeout would fire, which is the worst outcome: no decision from risk at all. With the cap, the scorer still has ~40 ms for rules.
   3. The scorer decides with rules only: platform rules, merchant rules that do not use the score, and a conservative segment policy. Rules only is not "allow everything": blocklists and card-testing rules still fire, and high-risk segments (a new card in a high-risk category above the merchant’s limit) get 3-D Secure or a decline, while the rest is allowed. With the numbers in the estimates it catches about half of the fraud the full model catches. It also runs in shadow on every payment, all the time, so its quality is measured and it is never cold when it is needed.
   4. Rules-only decisions are logged with their mode, and those payments are re-scored when the model is back, before capture. Payments decided by rules only are scored again from their logged features once the model recovers. Any above the review line go to the review queue before they are captured or shipped, which recovers much of what failing open let through. The share of payments decided in each mode is a top-line alert.
   5. An authorisation’s risk call is retried after a timeout; the scorer finds its decision key and returns the same answer. A merchant resending the same request with the same idempotency key is answered by the payment API without reaching risk. But the auth service can retry or hedge its own call, and the decision key makes that idempotent. Without it, the retry would be scored again and would see its own first attempt in the 1-minute counters, so the same payment could flip from allow to block between tries.
   6. If the scorer does not answer at all, the auth service applies its own static policy and goes on to the issuer. The auth service holds a tiny policy, refreshed daily: allow below the merchant’s limit, ask for 3-D Secure above it where available, decline above a hard cap in high-risk categories. A minute of degraded scoring at peak lets through roughly $50 k more fraud than the model would; failing closed would decline about $30 M of good payments in the same minute.
4. Review, a fraud report, then a chargeback. The asynchronous loop that teaches the system: a held payment is reviewed within the hour, another payment is reported as fraud by its issuer after nine days and charged back after 47, and each signal is joined to the decision it judges. Then the graph job turns the confirmed fraud into features for the cards and devices linked to it.
   1. A $900 order is scored into the review band; it is authorised but not captured, and the case enters the merchant’s review queue. Holding the capture is what makes review cheap: a rejected payment is voided before any money moves, so there is no chargeback. Priority is expected loss, p × amount, raised as the authorisation nears expiry (about 7 days), so a $900 order at 30 % comes before a hundred $20 orders at 40 %.
   2. An analyst opens the case with its linked entities and velocity history, rejects it, and the review service voids the authorisation. The case shows the reasons (new device, card used at 4 merchants today), the graph neighbourhood (this device has carried 11 cards, 3 of them disputed) and the customer’s history. Reject voids the hold; approve captures. Cases nobody reaches before their deadline get the expected-cost decision automatically, so queue length follows staffing and never blocks a payment indefinitely.
   3. The verdict becomes a label event, stamped with the time it became known. Reviewer labels arrive in hours but are biased: they exist only for the review band, and a rejected payment never gets a chargeback to confirm it. They are kept as their own label source so training can weight them, and reviewers’ accuracy is measured against the disputes that do arrive for approved cases.
   4. Nine days after a different payment, its issuer files a fraud report; on day 47 the chargeback arrives. The fraud report (Visa TC40, Mastercard SAFE) is the earliest reliable signal and usually arrives long before the dispute. The chargeback is final but slow: roughly half of fraud disputes come within 30 days and most within 90, with a window of about 120. Both are deduplicated by source id, and only fraud reason codes count, because "item not received" is a dispute, not fraud.
   5. The label updates the real-time features of that card, its devices and its email straight away. Confirmed fraud on a card fingerprint is a feature for the next payment on that card anywhere on the network, and for every device and email it has used. That goes through the velocity jobs in seconds; waiting for the next daily batch would leave a known stolen card usable for a day.
   6. The sink appends the label to the decision log, where it joins the decision logged 47 days earlier by attempt id. Nothing is recomputed: the training row is the feature vector exactly as served on day 0, plus the labels known by the training cutoff. Keeping known_at on every label is what lets an evaluation replay "what we knew on 1 March" honestly.
   7. The daily graph job links the disputed card to its devices, emails and addresses and writes ring features to the feature store. The card shares a device with 40 other cards, 12 of them disputed, and a shipping address with 6 more. Every entity in that component gets features such as component size, share of disputed cards and hops to the nearest confirmed fraud, so a brand-new card used on that device tomorrow scores high on its first payment. An hourly delta handles new links to confirmed fraud between daily runs.
5. Retrain, shadow and ramp a new model. A weekly retrain produces a model that is better offline. Getting it into production is the hard part, because the outcome it is judged on takes about 90 days to observe and blocked payments never get labels at all. Calibration, shadow scoring, a ramp by card and a long holdback replace the usual one-week A/B test.
   1. The training job builds a dataset from logged decisions and the labels known at the cutoff, with explored payments weighted by their propensity. Payments older than ~90 days carry dispute labels; newer ones carry issuer reports and review verdicts, and their unlabelled negatives are down-weighted by the share that historically still turns into chargebacks at that age. The few would-be blocks let through by exploration are weighted by 1 / propensity, so the model sees the region it would otherwise never get labels for.
   2. It calibrates the new model so each score means the same fraud rate as before, and backtests it against the live model on held-out weeks. Merchants have written rules such as "review if score > 65" tuned to the old scores; if the new model shifts the distribution, those rules silently change meaning. Calibrating to a fixed probability scale keeps score 65 at the same fraud rate. The backtest replays both models with every merchant’s thresholds on the same decisions: fraud caught, good payments declined, review volume.
   3. The new model is deployed to the model server as a shadow and scores every payment after the decision is returned. Shadow scoring never touches the latency budget: the scorer calls it with the same feature vector after it has replied. The shadow’s score, would-be decision and latency are logged next to the live ones for a week.
   4. A week of shadow logs compares decisions, score distributions per merchant segment and latency. The most useful view is the disagreement set: payments one model blocks and the other allows. Early labels on payments the old model allowed and the new one would block are the first evidence the new one is better; the decline rate on long-tenured good customers is the guardrail.
   5. The scorer ramps the new model by card fingerprint hash, 1 % to 10 % to 50 % to 100 %, checking guardrails at each step. Bucketing by card rather than by payment means a fraudster retrying a card cannot keep trying until it lands on the more lenient model. Guardrails per step: block, challenge and review rates per merchant segment, issuer approval rate on allowed payments, p99 latency. Each step bakes for a few days; rollback is a pointer flip to the previous version, which stays loaded.
   6. A 2 % holdback stays on the old model for 90 days, so the two models’ real chargeback rates can be compared when labels mature. Every comparison so far used proxies. The holdback is the only clean measurement of fraud loss and false declines between the two models, and it costs little: 2 % of traffic on a model that was good enough last week. Each decision logs which model made it, so the comparison is a query over the decision log in three months.

Deep dives

The theory behind it

• Real-time fraud detection: Designing fraud and risk systems for payments and marketplaces: the decision point and latency budget, rules plus machine learning, real-time velocity features, device and network signals, graph features, step-up actions, review queues, feedback and label delay, and measuring precision and losses.
• Feature stores and serving machine learning models: The infrastructure behind production machine learning: offline and online feature stores, training-serving skew, point-in-time correct training data, real-time features from streams, model serving with latency budgets, batching, shadow and canary deployments, and monitoring drift.
• Windowing, event time and watermarks: Aggregating streams correctly: event time versus processing time, tumbling, sliding and session windows, late and out-of-order events, watermarks and allowed lateness, triggers and updates, state size, and stream joins.

Related

• Design Search Autocomplete
• Design an Ad Click Aggregator
• Design a Stock Exchange
• Design a Project-to-Contractor Matching System
• All 83 system design interview questions
• Run this question for a friend: the interviewer kit