Design an A/B Testing Platform

• Define an experiment: Variants as sets of parameter values, a randomization unit (user, or device when logged out), a layer, a traffic share, a target audience (country, platform, app version, segments), a primary metric, guardrails, and a horizon taken from a power calculation.
• Assign consistently everywhere, with no network call: Every backend service and every app gets the same variant for the same unit, and a logged-in user keeps it across devices. A check costs microseconds and never depends on the platform being up.
• Run hundreds of experiments at once without interference: Experiments that change the same parameters must never share a user; experiments on different parameters should overlap so each gets enough traffic. Plus holdouts that see none of a quarter’s launches.
• Log exposure when the user actually sees the variant: At the point where the experiences diverge, in control and treatment alike, deduplicated. Not at assignment, which happens for many users who never reach the change.
• Compute metrics from exposures and outcome events: The outcome events (page views, purchases, errors) already flow through the company’s event pipeline. Join them to exposures per user: daily for every metric, near-real-time for guardrails.
• Results page with effects and confidence intervals: Per metric: the difference, relative effect, 95 % interval and p-value, variance-reduced. Warnings for sample ratio mismatch, results read before the planned horizon, many-metric false positives and underpowered tests.
• Guardrails with automatic shutdown: Crash rate, error rate, latency and revenue are attached to every experiment. A variant that clearly harms one is stopped within minutes, without waiting for its owner to notice.
• Change a running experiment safely: Ramp traffic without moving users already in, stop a variant, end and ship. Every change is versioned and audited, and a change that would move users between variants starts a new iteration instead.
• Out of scope: The product event pipeline itself (we consume it), feature flag management beyond sharing the SDK, bandits and automatic traffic allocation (see the bandits guide), switchback and cluster experiments for marketplaces and social products, and the metric authoring UI.

• Assignment cost (in-process · ~1 µs per check): No network call on the request path. At 2 M checks a second that is about two CPU cores across the whole fleet, against 2 M RPCs a second and a round trip on every check for a remote service.
• Consistency (same unit → same variant everywhere): Across services, devices for logged-in users, restarts and ramps. Determinism replaces stored state: nothing has to be looked up for the answer to be the same.
• Availability of assignment (never blocks · last snapshot, else defaults): A platform outage must never break a product page. SDKs keep serving the last snapshot they have; a process with none serves default parameters and logs no exposures, so it is invisible in the analysis.
• Config propagation (p99 < 60 s to 50 k processes): Starts, ramps and stops reach every process within a minute. Starts and ramps are scheduled a couple of minutes ahead so the whole fleet flips at the same instant; stops apply at once.
• Result freshness (daily by 06:00 UTC · guardrails < 15 min behind): Decision metrics change slowly and are read once a day. Guardrails exist to catch a broken variant, so they run on a separate streaming path that is minutes behind.
• Statistical trustworthiness (real false positive rate ≈ stated α): This is the requirement that drives the hardest trade-off. Teams want answers fast, look every morning and read 100 metrics, and each of those habits multiplies false positives. The platform holds the real rate near 5 % for every team at once, and checks sample ratios on every experiment.
• Scale (500 experiments · 300 M users · 50 B events/day): 100 M daily users, 2 M assignment checks a second at peak, and ~5 B exposures plus ~45 B outcome events a day.
• Auditability and privacy (every change versioned · deletion within 30 days): Who changed what and when, with the snapshot version stamped on every exposure, so any result can be reproduced. Per-user tables use pseudonymous ids and are purged on account deletion; the results page shows only aggregates.

• CPU for assignment, fleet-wide: ~2 cores. A check is two hashes and a few comparisons against an in-memory snapshot: ~1 µs. 2 M checks/s × 1 µs = 2 core-seconds per second, so about two cores across every service. A remote assignment service would instead take 2 M RPCs a second and put a network round trip on each of the dozens of checks in a request. Assignment cost is not the problem; distributing the config is.
• Config snapshot size: ~1 MB · ~200 KB compressed. 500 running experiments × ~2 KB each (layer, salt, bucket ranges, variant slots, parameters, compiled targeting) = ~1 MB, about 200 KB compressed. ~50 k server processes hold it, so a full push is 50 k × 200 KB = 10 GB; a delta for one changed experiment is ~2 KB, or ~100 MB fleet-wide.
• Events per second: ~580 k/s avg · ~1.5 M/s peak. 50 B events/day ÷ 86 400 ≈ 580 k/s on average, and roughly 2.5× that at the evening peak: ~1.5 M/s. About 5 B of them are exposures; the other ~45 B are the product events that already flow through the company pipeline.
• Exposures logged per day: ~5 B (vs ~86 B raw checks). 100 M daily users × ~20 experiments each is exposed to × ~2.5 sessions, logging once per user, experiment and session = 5 B/day. Logging every check instead would be ~1 M checks/s average × 86 400 ≈ 86 B/day, seventeen times more, which is why the SDK deduplicates in the process.
• Raw event storage: ~10 TB/day · ~0.7 PB/year compressed. 50 B events × ~200 B = 10 TB/day raw, about 2 TB/day in compressed columnar files. A year is ~0.7 PB. Most of this exists already for product analytics; the platform adds the exposures table, ~0.5 TB/day raw.
• Daily join volume: ~2 B rows (vs ~900 B). Roll events up to one row per user per day first: ~100 M rows. Join those to each user’s ~20 running experiments: 100 M × 20 = 2 B rows/day. Joining raw events instead would be 45 B × 20 = 900 B rows, 450 times more. Aggregating per user before the join is the whole trick.
• Users per arm for a 1 % lift: ~3 M · ~1.5 M with CUPED. n ≈ 16 σ² / δ² per arm at α = 0.05 two-sided and 80 % power. For 5 % conversion σ² = 0.05 × 0.95 = 0.0475, and a 1 % relative lift is δ = 0.0005: 16 × 0.0475 / 0.00000025 ≈ 3.0 M. CUPED with a pre-period correlation of 0.7 removes about half the variance: ~1.5 M.
• Users per experiment if none overlap: ~600 k. 300 M users ÷ 500 experiments = 600 k each, about 300 k per arm, which can only detect lifts of ~3 % on a 5 % conversion rate. Most real wins are under 1 %. Experiments have to share users, so the question becomes how they share them without interfering.

• Apps (web, mobile) (cached assignments): The iOS, Android and web clients. At session start they fetch this user’s client-side assignments from the edge, about 2 KB, cache them with a TTL and read parameters from that cache synchronously, so no screen waits on the network. They log an exposure from the same call that reads a parameter, where the experience diverges, in control as well as in treatment.
• Edge API (assignments · event intake): The front door for app traffic. Serves GET /v1/assignments by running the same SDK over its in-memory snapshot for this user, or this device when logged out, so phones never download the experiment config and unreleased features never ship inside an app. Also takes the apps’ event batches, produces them to the event log, and forwards product requests to the services.
• Product services (in-process SDK): Every backend service (search, ranking, checkout, pricing) embeds the SDK. A check hashes the unit id against the in-memory snapshot in about a microsecond, returns the variant’s parameters and queues an exposure, deduplicated per (unit, experiment) for 30 minutes inside the process. With no snapshot at all, the SDK returns default parameters and logs nothing.
• Config snapshots (CDN · immutable versions): Compiled, signed, immutable snapshot files (~1 MB, ~200 KB compressed) and deltas between consecutive versions (~2 KB for one changed experiment), cached on a CDN in front of object storage. SDKs long-poll a tiny latest-version pointer and fetch only what changed, so a change reaches 50 k processes as ~100 MB of deltas rather than 10 GB of snapshots. Every SDK also keeps the last good snapshot on local disk for restarts.
• Guardrail monitor (stream · 5-min windows): A streaming job over exposures and guardrail events (crashes, errors, latency, checkout failures), both keyed by unit id. Keeps each user’s exposures from the last 24 h in keyed state, ~32 GB across the cluster, counts guardrail outcomes per variant in 5-minute windows and runs an always-valid sequential test, so looking 288 times a day does not manufacture alarms. On a clear, large harm it stops the variant through the experiment service; it can never ship or restart anything.
• Event log (Kafka · keyed by unit id): The company’s existing log carries ~45 B outcome events a day; the platform adds an exposures topic of ~5 B a day. Both are keyed by unit id, so a user’s exposures and events land on the same partition, which lets the guardrail monitor join them without a shuffle. Retained for days, so a broken consumer can rewind instead of losing data.
• Experiment console (design · ramp · results): Where product teams design an experiment, run the power calculation, ramp, stop and read results. It enforces the plan: a primary metric, guardrails and a horizon must be chosen before start, and the decision badge of a fixed-horizon test stays “not final” until that date however green the numbers look.
• Experiment service (validate · allocate · audit): Owns every experiment’s lifecycle (draft, running, stopped, ended) and every change to it. Validates definitions (each parameter belongs to exactly one layer, metrics exist, guardrails attached), allocates layer buckets in a transaction so two experiments in a layer can never share one, and compiles and publishes a new snapshot version on each change. Refuses changes that would move users between variants and offers a new iteration with a fresh salt instead.
• Config store (Postgres · versioned): Experiments and their iterations, layers with their 10,000 buckets and each bucket’s owner, the metric catalog, and an append-only history of config versions with actor, reason and activation time. Megabytes of live state, strongly consistent because bucket ownership is a mutual-exclusion invariant. The pipeline reads the allocation history from here to know the expected split on every day.
• Event warehouse (Parquet · by day): All exposures and outcome events in columnar files partitioned by event date and hour: ~10 TB a day raw, ~2 TB compressed. Also holds the pipeline’s derived tables: user-day rollups, kept 90 days so every experiment’s pre-period is available for CUPED, first exposures per experiment, and per-user running totals.
• Stats engine (CIs · CUPED · SRM): Turns sums into decisions. Runs the sample ratio check first and replaces the scorecard with a warning if it fails, then computes CUPED-adjusted differences, relative effects with delta-method intervals, fixed-horizon or sequential significance per the experiment’s plan, and false discovery control across secondary metrics. Also answers power calculations at design time from stored baselines.
• Results store (per-variant sums): Sufficient statistics per (experiment, iteration, variant, metric, day, slice): n, Σx, Σx² and the CUPED and ratio cross-terms, plus per-metric baselines for power calculations. About 150 k rows a day before slices and ~3 M with them, small enough that any interval is recomputed on request. Never holds per-user data.
• Metrics pipeline (Spark · daily): Runs after each day’s partitions close. Rolls 45 B outcome events into one row per user per day, shared by all 500 experiments, joins those rows to first exposures, keeps per-user running totals from first exposure, and reduces them to per-variant sufficient statistics. Everything after it is arithmetic on a few hundred thousand rows; the run takes about two hours.

1. A user gets their variants and is exposed: The hot path. Assignment is computed locally wherever it is needed, in microseconds, and the only thing that leaves the process is an exposure log, written asynchronously at the moment the user actually sees the change.
   1. The app starts a session and fetches this user’s client-side assignments from the edge.
   2. The edge evaluates every eligible experiment for the user from its in-memory snapshot.
   3. The user opens checkout; reading the parameter logs an exposure, which goes up in the app’s next event batch.
   4. The checkout request reaches the pricing service, which checks a server-side experiment in-process.
   5. The service sends the exposure, and later the purchase, to the event log off the request path.
   6. Exposures and product events are written to the warehouse, partitioned by day.
2. Launching an experiment: The write path. A definition becomes buckets in a layer and a new snapshot version, and every process starts the experiment at the same instant without anyone calling the platform on the request path.
   1. The owner designs the experiment and asks how many users and days it needs.
   2. The experiment service validates the definition and stores it as a draft.
   3. On start, the service takes free buckets in the layer and records a new config version, in one transaction.
   4. The service compiles a snapshot and publishes it with a delta from the previous version.
   5. Every SDK picks up the new version and swaps it in, ready for the scheduled start.
   6. Exposures start to arrive, and the guardrail monitor begins watching the new experiment.
3. Daily results for 500 experiments: The scale case. Fifty billion events a day become a few hundred thousand sums, because events are aggregated per user once and shared by every experiment, and everything after that is arithmetic.
   1. After the day’s partitions close, the pipeline loads the running experiments and the metric definitions.
   2. One pass rolls the day’s 45 B outcome events into one row per active user.
   3. First exposures are updated for every experiment and user.
   4. The day’s user rows are joined to first exposures and added to each user’s running totals.
   5. Per-user totals are reduced to a few sums per variant and written to the results store.
   6. The stats engine checks the sample ratio first, then computes effects and intervals.
   7. The owner opens the results page and sees where the experiment stands against its plan.
4. A variant raises crashes: automatic stop: The failure path. A daily pipeline is far too slow when a treatment breaks the app, so a streaming monitor watches guardrails with a test that stays valid when checked every five minutes, and a stop reaches every process in under a minute.
   1. Android users in an image-cache experiment’s treatment start crashing, and the crash reports arrive with their exposures.
   2. The guardrail monitor attributes each crash to the experiments the user was exposed to, and counts per variant.
   3. Every five minutes it runs a sequential test on each guardrail, for each variant against control.
   4. The monitor asks the experiment service to stop the harmful variant.
   5. The service commits the change and publishes it on the urgent path, with no scheduled delay.
   6. SDKs swap in the new version, and the apps refresh their assignments.
   7. The owner is paged with the evidence, and a fixed version must start as a new iteration.
5. Ramping from 5 % to 50 % without moving anyone: Changing a running experiment. Widening the traffic must add users without moving any who are already in, and a change that would move them is refused rather than quietly applied.
   1. After a clean first day, the owner ramps the experiment from 5 % to 50 % of the layer.
   2. The service extends the experiment’s bucket range in the layer, taking only free buckets.
   3. The new version reaches every process and activates at the same instant.
   4. Users already in keep their variant, and newly included users split evenly.
   5. The owner then asks to change the split to 20/80, and the service refuses to apply it in place.
   6. A new iteration starts with a fresh salt, and its data is analysed separately.
   7. The pipeline reads the allocation history, so each day’s expected split is known.

Ask: Why not have every app and service call an assignment service, and store each user’s variant so it never changes?

Good answers name: Deterministic hash in an in-process SDK, evaluated from a pushed snapshot, Central assignment service with a stored variant per user, Random assignment remembered in a cookie or on the device, Ship the full snapshot to every app and evaluate on the device.

Our pick: Every check is a pure function. The SDK hashes (layer salt, unit id) with a fast, well-mixed hash such as MurmurHash3 into one of 10,000 buckets, finds the experiment that owns that bucket in the layer, evaluates its targeting against attributes the caller passed, then hashes (experiment salt and iteration, unit id) into 1,000 variant slots against the weights. The config is a compiled, signed, versioned snapshot of about 1 MB that every process holds in memory and refreshes from a CDN with ~2 KB deltas, p99 under 60 s, and starts are scheduled a couple of minutes ahead so every process flips at the same instant. With no snapshot at all, the SDK returns default parameters and logs no exposure, so a failure falls back to the control experience and stays out of the analysis. Apps get their assignments from the edge, which runs the same SDK for the user and returns only that user’s variants. Every SDK passes a shared conformance suite of (salt, id) → bucket vectors, because two languages that hash a string differently would silently split one user in two.

1. A user browses logged out, then logs in halfway through the session. Which variant do they get?
   It depends on the experiment’s unit, which the owner must choose. A user-unit experiment cannot be evaluated without a user id, so logged-out traffic gets the defaults and logs no exposure; after login the user is assigned by user id and keeps that variant on every device. A device-unit experiment, such as the signup page, keeps hashing the device id after login, so the experience does not change mid-session. What you must never do is switch the hash input at login within one experiment, because that moves users between arms.
2. Two SDKs in different languages disagree for 0.1 % of users. How would you find out?
   From the data, before anyone reports it. A unit seen in two variants of the same experiment is impossible under correct hashing, so the pipeline counts those per experiment and per SDK version and alerts above a tiny rate. The second symptom is a sample ratio mismatch that appears on one platform only. Prevention is a conformance suite of salts and ids with their expected buckets, including non-ASCII ids, that every SDK runs in CI.
3. The experiment service is down for an hour. What breaks?
   Nothing a user can see. SDKs keep serving their current snapshot, and new processes start from the copy on local disk or from the last version on the CDN, neither of which needs the service. What you lose is change: no launches, no ramps and, most importantly, no stops through the normal path. So stopping has a break-glass route: an on-call tool writes a signed override file next to the snapshots, and SDKs apply it on top of their current version.
4. Why 10,000 buckets in a layer and not 100?
   Granularity and room. With 100 buckets the smallest allocation is 1 % of the layer, and a layer holds at most 100 experiments; 10,000 gives 0.01 % steps for cautious first ramps and space to place experiments without fragmenting the layer. It costs nothing, because the snapshot stores ranges of buckets rather than lists, so a layer is a few dozen ranges whatever the count.
5. How do you know the assignment is actually random?
   Run A/A experiments continuously: two identical variants analysed exactly like real ones, through the same SDK, logging and pipeline. Across many A/A tests the p-values should be uniform, with about 5 % below 0.05, and every one should pass the sample ratio check. If not, something in hashing, exposure logging or the statistics is wrong, and you learn it before a real decision depends on it.

Ask: Five hundred experiments share 300 M users. Does each get its own slice of users or do they overlap, and how do you stop them interfering?

Good answers name: An exclusive slice of users for every experiment, Full overlap, every experiment hashed independently, Layers: exclusive within a layer, orthogonal across layers, domains for holdouts, Full factorial design with the interaction analysed.

Our pick: Parameters are registered in exactly one layer (checkout_ui, search_ranking, pricing, notifications…), and validation rejects an experiment that sets a parameter outside its own layer. Each layer has its own salt and 10,000 buckets; an experiment owns bucket ranges assigned by the experiment service in a transaction, so ownership never overlaps, and splits its users into variants with a second hash using its own salt. Before any layer, the SDK checks the domains: a global holdout of 2 % of users per product area per quarter, hashed with its own salt, gets every layer’s defaults and no experiments. Because layer salts are independent, experiments in different layers are crossed evenly, each arm of one containing a proportional mix of the other’s arms, so main effects stay unbiased unless the two genuinely interact. The platform measures the pairs that could: same surface, different layers, overlapping in time.

1. How would you detect that two experiments in different layers interact?
   For each pair running at the same time on the same surface, the pipeline already has users in all four combinations, so it estimates the 2 × 2 interaction term per metric and flags pairs where it is both significant and large. With hundreds of pairs, apply false discovery control or you will chase noise. True interactions turn out to be rare; when one is real, move the two experiments into one layer or run them as a factorial design.
2. The search ranking layer is full: every bucket is owned. A team wants to start today.
   Then they queue, which is the right answer for parameters that genuinely collide. Make the queue shorter: end experiments that reached their horizon, size experiments with the power calculation instead of taking 50 % by habit, and split the layer if it holds parameters that do not really interact, since the ranking model and snippet length can be two layers. The console shows each layer’s occupancy and when buckets will be released, so teams can plan.
3. Buckets released by one experiment go to the next. Are the new experiment’s users contaminated by what they saw before?
   They carry the previous experience, but evenly. The new experiment splits users with its own salt, independent of the old split, so both new arms contain the same mix of old control and old treatment users, and the comparison stays unbiased. A strong lingering effect still adds noise, so the service prefers buckets that have been free longest, and a restarted experiment always gets a new salt rather than reusing the old split.
4. Every experiment this quarter shipped a 1 % win on conversion. Did conversion rise 30 %?
   Almost certainly not, and the global holdout is how you find out. Shipped wins are overestimates because only the ones that came out significant were shipped, the winner’s curse; novelty fades; and some launches improve the same thing. The holdout, 2 % of users who got none of the quarter’s launches, is compared with everyone else at the end of the quarter, and with 6 M held-out users it detects a cumulative change of about half a percent. It costs those users the improvements for a quarter, which is why it is small and time-boxed.
5. A team needs to change parameters in two layers within one experiment.
   Then those parameters interact by definition, and the layering is wrong for them. In order of preference: move the parameters into one layer; or run the experiment in its own domain, a slice of traffic with its own layering; or redesign the layers. Letting one experiment span two layers would break the exclusivity guarantee for every other experiment in both.

Ask: Why not log every user at the moment they are bucketed, and analyse everyone assigned?

Good answers name: Log at assignment, analyse everyone assigned, The getter logs exposure at the divergence point, in every arm, Log exposure when the treatment UI renders, Stamp every outcome event with the user’s current assignments.

Our pick: The SDK’s get() is the only way to read an experiment’s parameters, and it logs the exposure, so the log happens where the code reads the value, in every arm. Teams are told to call it at the divergence point (rendering the checkout page, calling the ranker), and a lint rule flags calls at request entry. Exposures carry unit, experiment, iteration, variant, snapshot version and time; they are deduplicated per (unit, experiment) for 30 minutes in each process and per session on devices, and go to the exposures topic keyed by unit id: ~5 B a day instead of ~86 B raw checks. The analysis counts a user from their first exposure. The sample ratio check runs on exposure counts every day, a chi-squared test against the configured split, and below p = 0.0005 it hides the results behind a warning.

1. The treatment page is 300 ms slower, so more treatment users leave before the exposure fires. What happens?
   Exactly the bias the sample ratio check exists to catch: fewer treatment exposures than the split says, and the treatment users who remain are the patient ones, which flatters the treatment. Logging at the getter, before rendering, mostly removes it because the decision is logged before the slow part. If the check still fails, the result is not trustworthy; the remedy is to fix the logging point and rerun, not to re-weight the arms.
2. The split is 1,000,000 against 990,000. A 1 % gap. Does it matter?
   Yes, a great deal. Against an expected 995,000 each, the chi-squared statistic is about 50, a p-value around 10⁻¹², so this is not chance. Something is losing or adding users in one arm, and whatever it is probably correlates with the outcome: bot filtering that reacts to the treatment, a redirect that drops events, a client version that fails to log. A 1 % imbalance looks small to a person and is enormous to the test.
3. A prefetch evaluates the checkout experiment for users who never open checkout.
   Then exposure is over-counted equally in both arms, which brings back some dilution but no bias. The fix is an SDK call that evaluates without logging, for prefetching and caching, plus an explicit log call when the prefetched page is actually shown. Keeping the default get() logging and the silent call the exception keeps the common case right.
4. How do you make sure control logs at the same point as treatment?
   Make it hard to do anything else: both arms read the same parameter through the same get() call, and the code branches on the returned value afterwards. What breaks symmetry is code that asks “am I in treatment” through a separate path, or reads the experiment only inside the new component, so review guidelines and a lint rule look for exactly that. The sample ratio check is the backstop.
5. Why put the snapshot version on every exposure?
   So every number can be traced to the config that produced it. If a ramp or a stop happened during the day, the pipeline knows which allocation each exposure was made under, which is how it knows the expected split for the sample ratio check and how it spots a process stuck on an old snapshot. It costs four bytes an event.

Ask: How do you compute 100 metrics for 500 experiments every day without scanning the event firehose 500 times?

Good answers name: One query per experiment over the raw events, Shared user-day rollup, joined to first exposures, reduced to per-variant sums, Streaming computation for every metric, Query the warehouse on demand when the page is opened.

Our pick: A daily Spark job runs once the day’s partitions close. It rolls the day’s 45 B outcome events into user_day, one row per active user (~100 M) with a column per catalog measure. It updates first_exposure per (experiment iteration, unit) with the variant and time. It joins the day’s user_day rows to first exposures, ~2 B rows, and adds them into per-user running totals since first exposure; on the first day only events after the exposure time count, taken from raw events for just the newly exposed users. Finally it reduces those totals per (experiment, variant, metric, slice) to n, Σx and Σx², the covariate sums Σc, Σc² and Σxc for CUPED, and for ratio metrics the numerator, denominator and cross-product sums for the delta method. It takes about two hours, so results are ready by 06:00 UTC. Percentile metrics such as p95 latency keep a mergeable quantile sketch per variant instead of sums.

1. Click-through rate is clicks over page views. Why not compute it per page view and run a t-test on that?
   Because page views are not independent: one heavy user contributes hundreds of correlated views, so treating views as the sample overstates the sample size and makes intervals too narrow, which means false positives. Randomisation was by user, so compute each user’s clicks and views, take the ratio of sums per variant, and get its variance with the delta method from Σclicks, Σviews, their squares and their cross-product. Same point estimate, honest interval.
2. Someone finds a bug in the revenue metric after an experiment has run for ten days.
   Metric definitions are versioned, so the fix is a new version plus a backfill: recompute that metric’s per-user totals from user_day for the experiment’s days, which is cheap because the rollup is already per user, and rewrite the per-variant sums. The results page shows which definition version produced each number. Never edit the old version in place, or yesterday’s screenshots stop matching anything.
3. A phone was offline for three days and uploads its events late.
   They land in the partitions for their event dates when they arrive, so every run recomputes a trailing window, say the last three days, not only yesterday. Anything later is picked up by a weekly recompute. Recent days are labelled as still settling, the same idea as the ad click aggregator’s provisional days.
4. How big is the per-user running-total table, and why keep it?
   About 4 B (experiment, user) rows across running experiments, each with that experiment’s metric totals and covariates: a few terabytes, rewritten incrementally. It makes the daily job additive, since today’s totals are yesterday’s plus one day, so the job never rescans an experiment’s whole history. It is also where per-user rules apply, such as capping each user’s revenue before it is summed.
5. A VP wants results every hour, not every day.
   For guardrails that already exists, on the streaming path. For decision metrics, hourly results mostly add more looks at the same data, which makes peeking worse, and most metrics have weekly cycles anyway. If the business truly needs it, run the same pipeline incrementally over hourly partitions for a few key metrics, but keep the decision tied to the planned horizon.

Ask: Teams check results every morning and the scorecard has 100 metrics. How does the platform keep the false positive rate anywhere near 5 %?

Good answers name: Fixed horizon: one decision at the planned end, interim numbers marked not final, Always-valid sequential tests (mSPRT-style confidence sequences), Group sequential design with a few planned looks (O’Brien-Fleming), Report a Bayesian probability to beat control instead of p-values.

Our pick: Every experiment declares its analysis plan at start: a primary metric, a minimum detectable effect, a horizon from the power calculation rounded up to whole weeks, and fixed-horizon or sequential. Fixed horizon is the default: interim results are visible, the decision badge says not final until the horizon, and shipping early is recorded against the owner. Guardrails always use always-valid sequential tests, because the monitor looks every 5 minutes, and teams that need early stopping on the primary metric opt into a group sequential plan with up to five planned looks, whose boundaries are stored with the plan. For many metrics, the primary metric is tested at α = 0.05, guardrails are tested only for harm, and the rest of the scorecard and every slice by country or platform are corrected with Benjamini-Hochberg at a 5 % false discovery rate and labelled exploratory, to be confirmed by a follow-up experiment. An implausibly large effect gets a Twyman’s law warning, as the Kohavi, Tang and Xu book advises: check the logging before celebrating.

1. The guardrail monitor stops a variant that was actually fine. What did that cost, and how do you keep it rare?
   A stopped variant and an unhappy team, because the experiment loses its treatment users and needs a new iteration. Keep it rare in two ways: the sequential test runs at a low α such as 0.001 however many looks it takes, and an automatic stop also requires a practically large harm, such as a crash-rate interval entirely above +10 %, so a significant but trivial difference only alerts a person. Then measure it: false stops per quarter, from A/A experiments and owner appeals.
2. Why not just apply Bonferroni across all 100 metrics?
   Because it controls the chance of even one false positive among 100, with a threshold near 0.0005 per metric, which makes the scorecard nearly blind and real secondary effects vanish. False discovery control asks a more useful question, what share of the flagged metrics are false, and keeps far more power. Bonferroni is reasonable for a handful of guardrails, where crying wolf is the cost to avoid.
3. The effect was +4 % in week one and +1 % in week two. Which is real?
   Probably neither on its own: that is the shape of a novelty effect, people clicking on something new, and the long-run effect may be lower still. The results page plots the effect by day since first exposure for exactly this reason. Run to the horizon, and for launches where the long run matters, keep a 5 % holdback on the old experience for a month and read it then.
4. A team runs four variants against one control. Does anything change?
   Yes: four comparisons on the primary metric are four chances at a false positive, so the stats engine adjusts for them, with Holm’s method for example, and the power calculation splits traffic five ways rather than two. Every extra variant costs traffic, so the console shows the minimum detectable effect getting worse as variants are added.
5. How do you prove the platform’s statistics are right?
   Run A/A tests constantly through the full production path (same SDK, logging, pipeline and stats engine) and check that their p-values are uniform, that about 5 % fall below 0.05, that sequential tests stop falsely at their stated rate, and that the sample ratio check passes. Replaying past A/A data with simulated effects of known size checks power as well. The platform should be held to the standard it holds product teams to.

Ask: A 1 % lift needs 3 M users per arm. How do you get the same answer with fewer users or in less time?

Good answers name: CUPED: adjust by the same metric over the 14 days before the experiment, Cap outliers before summing (winsorise), Run longer, Switch to a more sensitive proxy metric.

Our pick: For each metric the pipeline computes the same metric per user over the 14 days before the experiment started, from user_day, which is kept 90 days, and stores Σc, Σc² and Σxc per variant beside n, Σx and Σx². The stats engine estimates θ = cov(x, c) / var(c) pooled across arms and reports the adjusted difference, the difference in means minus θ times the difference in pre-period means, with its variance from the same sums. Users with no pre-period get c = 0, which keeps the estimate unbiased but gives them no reduction. Revenue-like metrics are also capped at a 99.9th percentile fixed from pre-experiment data. At ρ = 0.7, 3 M users per arm become about 1.5 M, and the power calculation already accounts for it, so experiments are planned at the shorter length. The pre-period means double as a check: if they differ significantly between arms, the randomisation or the logging is suspect.

1. Doesn’t adjusting the metric with another variable let you manufacture a result?
   Not if the covariate is fixed before the experiment and chosen before the analysis. The pre-period value cannot be affected by the treatment, so its expected difference between arms is zero, and subtracting θ times that difference changes the variance, not the expected effect. Choosing the covariate after seeing the results would be manipulation, which is why the platform fixes it per metric: the same metric over the 14 pre-experiment days.
2. Most users in a signup experiment are brand new. Does CUPED help?
   Hardly. New users have no pre-period, so their covariate is zero and they get no reduction. Other pre-assignment information still helps a little, such as platform, country or acquisition channel, used as covariates in a regression adjustment, which generalises CUPED. For new-user experiments the honest answer is more traffic or more time.
3. Why 14 days of pre-period, not 90?
   Diminishing returns and drift. The correlation with the in-experiment metric rises quickly over the first week or two and then flattens, while a longer window means more history to keep and more users whose behaviour has changed. Two weeks also covers two full weekly cycles. It is a per-metric setting, so a metric with a monthly rhythm can use more.
4. How do you apply CUPED to a ratio metric like click-through rate?
   Linearise first. With the delta method, a ratio of sums behaves approximately like a mean of a per-user linear combination of numerator and denominator, so the adjustment is built on that linearised value, with the pre-period numerator and denominator as covariates. It all still comes from per-variant sums, a few more cross-products. More bookkeeping, not a different method.
5. How would you know how much CUPED is saving?
   Measure it instead of assuming: the stats engine records the variance with and without the adjustment for every experiment and metric, so the platform reports the reduction per metric. Engagement metrics with stable users gain the most and conversion metrics with many new users the least. Converted into experiment-days saved across 500 experiments, it is usually the single most valuable statistical feature a platform ships.