System design interview question

Design an A/B Testing Platform

Q: Why not have every app and service call an assignment service, and store each user’s variant so it never changes?

Every check is a pure function. The SDK hashes (layer salt, unit id) with a fast, well-mixed hash such as MurmurHash3 into one of 10,000 buckets, finds the experiment that owns that bucket in the layer, evaluates its targeting against attributes the caller passed, then hashes (experiment salt and iteration, unit id) into 1,000 variant slots against the weights. The config is a compiled, signed, versioned snapshot of about 1 MB that every process holds in memory and refreshes from a CDN with ~2 KB deltas, p99 under 60 s, and starts are scheduled a couple of minutes ahead so every process flips at the same instant. With no snapshot at all, the SDK returns default parameters and logs no exposure, so a failure falls back to the control experience and stays out of the analysis. Apps get their assignments from the edge, which runs the same SDK for the user and returns only that user’s variants. Every SDK passes a shared conformance suite of (salt, id) → bucket vectors, because two languages that hash a string differently would silently split one user in two.

Q: Five hundred experiments share 300 M users. Does each get its own slice of users or do they overlap, and how do you stop them interfering?

Parameters are registered in exactly one layer (checkout_ui, search_ranking, pricing, notifications…), and validation rejects an experiment that sets a parameter outside its own layer. Each layer has its own salt and 10,000 buckets; an experiment owns bucket ranges assigned by the experiment service in a transaction, so ownership never overlaps, and splits its users into variants with a second hash using its own salt. Before any layer, the SDK checks the domains: a global holdout of 2 % of users per product area per quarter, hashed with its own salt, gets every layer’s defaults and no experiments. Because layer salts are independent, experiments in different layers are crossed evenly, each arm of one containing a proportional mix of the other’s arms, so main effects stay unbiased unless the two genuinely interact. The platform measures the pairs that could: same surface, different layers, overlapping in time.

Q: Why not log every user at the moment they are bucketed, and analyse everyone assigned?

The SDK’s get() is the only way to read an experiment’s parameters, and it logs the exposure, so the log happens where the code reads the value, in every arm. Teams are told to call it at the divergence point (rendering the checkout page, calling the ranker), and a lint rule flags calls at request entry. Exposures carry unit, experiment, iteration, variant, snapshot version and time; they are deduplicated per (unit, experiment) for 30 minutes in each process and per session on devices, and go to the exposures topic keyed by unit id: ~5 B a day instead of ~86 B raw checks. The analysis counts a user from their first exposure. The sample ratio check runs on exposure counts every day, a chi-squared test against the configured split, and below p = 0.0005 it hides the results behind a warning.

Q: How do you compute 100 metrics for 500 experiments every day without scanning the event firehose 500 times?

A daily Spark job runs once the day’s partitions close. It rolls the day’s 45 B outcome events into user_day, one row per active user (~100 M) with a column per catalog measure. It updates first_exposure per (experiment iteration, unit) with the variant and time. It joins the day’s user_day rows to first exposures, ~2 B rows, and adds them into per-user running totals since first exposure; on the first day only events after the exposure time count, taken from raw events for just the newly exposed users. Finally it reduces those totals per (experiment, variant, metric, slice) to n, Σx and Σx², the covariate sums Σc, Σc² and Σxc for CUPED, and for ratio metrics the numerator, denominator and cross-product sums for the delta method. It takes about two hours, so results are ready by 06:00 UTC. Percentile metrics such as p95 latency keep a mergeable quantile sketch per variant instead of sums.

Q: Teams check results every morning and the scorecard has 100 metrics. How does the platform keep the false positive rate anywhere near 5 %?

Every experiment declares its analysis plan at start: a primary metric, a minimum detectable effect, a horizon from the power calculation rounded up to whole weeks, and fixed-horizon or sequential. Fixed horizon is the default: interim results are visible, the decision badge says not final until the horizon, and shipping early is recorded against the owner. Guardrails always use always-valid sequential tests, because the monitor looks every 5 minutes, and teams that need early stopping on the primary metric opt into a group sequential plan with up to five planned looks, whose boundaries are stored with the plan. For many metrics, the primary metric is tested at α = 0.05, guardrails are tested only for harm, and the rest of the scorecard and every slice by country or platform are corrected with Benjamini-Hochberg at a 5 % false discovery rate and labelled exploratory, to be confirmed by a follow-up experiment. An implausibly large effect gets a Twyman’s law warning, as the Kohavi, Tang and Xu book advises: check the logging before celebrating.

Q: A 1 % lift needs 3 M users per arm. How do you get the same answer with fewer users or in less time?

For each metric the pipeline computes the same metric per user over the 14 days before the experiment started, from user_day, which is kept 90 days, and stores Σc, Σc² and Σxc per variant beside n, Σx and Σx². The stats engine estimates θ = cov(x, c) / var(c) pooled across arms and reports the adjusted difference, the difference in means minus θ times the difference in pre-period means, with its variance from the same sums. Users with no pre-period get c = 0, which keeps the estimate unbiased but gives them no reduction. Revenue-like metrics are also capped at a 99.9th percentile fixed from pre-experiment data. At ρ = 0.7, 3 M users per arm become about 1.5 M, and the power calculation already accounts for it, so experiments are planned at the shorter length. The pre-period means double as a check: if they differ significantly between arms, the randomisation or the logging is suspect.

Run 500 experiments at once on 300 M users: sticky variants in microseconds, exposure logging, and confidence intervals teams can trust.

Last updated 2026-10-26. Difficulty: hard. Patterns: experimentation, hashing, statistics, stream-processing, analytics. Reported at Meta, Netflix, Airbnb, Uber, Microsoft, LinkedIn, Booking.com.

Walk through a strong candidate's answer, turn by turn.

The interviewer asks, the candidate answers and draws, and you press Next. Skip to any stage or key moment, and ask the AI Mentor anything along the way.

Walk through it Take the mock interview

Or sit it in the style of Meta · Netflix · Airbnb · Uber, run as that company’s round is reported to go.

The short answer

How to design A/B Testing Platform in a system design interview, in five steps:

Scope it. Define an experiment, assign consistently everywhere, with no network call, run hundreds of experiments at once without interference, log exposure when the user actually sees the variant, compute metrics from exposures and outcome events, results page with effects and confidence intervals, guardrails with automatic shutdown, change a running experiment safely.
Set the targets. Assignment cost: in-process · ~1 µs per check; Consistency: same unit → same variant everywhere; Availability of assignment: never blocks · last snapshot, else defaults; Config propagation: p99 < 60 s to 50 k processes.
Do the numbers. CPU for assignment, fleet-wide: ~2 cores; Config snapshot size: ~1 MB · ~200 KB compressed; Events per second: ~580 k/s avg · ~1.5 M/s peak.
Draw the design. Edge API, Product services, Config snapshots, Guardrail monitor, Event log, Experiment service, Config store, Event warehouse, Stats engine.
Go deep where it is hard.
- Assignment without a network call → Deterministic hash in an in-process SDK, evaluated from a pushed snapshot.
- Overlapping experiments: layers and holdouts → Layers: exclusive within a layer, orthogonal across layers, domains for holdouts.
- Logging exposure, not assignment → The getter logs exposure at the divergence point, in every arm.
- From 50 B events to a confidence interval → Shared user-day rollup, joined to first exposures, reduced to per-variant sums.
- Peeking, sequential tests and many metrics → Fixed horizon: one decision at the planned end, interim numbers marked not final.
- Variance reduction with CUPED → CUPED: adjust by the same metric over the 14 days before the experiment.

Functional requirements

Define an experiment. Variants as sets of parameter values, a randomization unit (user, or device when logged out), a layer, a traffic share, a target audience (country, platform, app version, segments), a primary metric, guardrails, and a horizon taken from a power calculation.
Assign consistently everywhere, with no network call. Every backend service and every app gets the same variant for the same unit, and a logged-in user keeps it across devices. A check costs microseconds and never depends on the platform being up.
Run hundreds of experiments at once without interference. Experiments that change the same parameters must never share a user; experiments on different parameters should overlap so each gets enough traffic. Plus holdouts that see none of a quarter’s launches.
Log exposure when the user actually sees the variant. At the point where the experiences diverge, in control and treatment alike, deduplicated. Not at assignment, which happens for many users who never reach the change.
Compute metrics from exposures and outcome events. The outcome events (page views, purchases, errors) already flow through the company’s event pipeline. Join them to exposures per user: daily for every metric, near-real-time for guardrails.
Results page with effects and confidence intervals. Per metric: the difference, relative effect, 95 % interval and p-value, variance-reduced. Warnings for sample ratio mismatch, results read before the planned horizon, many-metric false positives and underpowered tests.
Guardrails with automatic shutdown. Crash rate, error rate, latency and revenue are attached to every experiment. A variant that clearly harms one is stopped within minutes, without waiting for its owner to notice.
Change a running experiment safely. Ramp traffic without moving users already in, stop a variant, end and ship. Every change is versioned and audited, and a change that would move users between variants starts a new iteration instead.
Out of scope. The product event pipeline itself (we consume it), feature flag management beyond sharing the SDK, bandits and automatic traffic allocation (see the bandits guide), switchback and cluster experiments for marketplaces and social products, and the metric authoring UI.

Non-functional requirements

Assignment cost (in-process · ~1 µs per check). No network call on the request path. At 2 M checks a second that is about two CPU cores across the whole fleet, against 2 M RPCs a second and a round trip on every check for a remote service.
Consistency (same unit → same variant everywhere). Across services, devices for logged-in users, restarts and ramps. Determinism replaces stored state: nothing has to be looked up for the answer to be the same.
Availability of assignment (never blocks · last snapshot, else defaults). A platform outage must never break a product page. SDKs keep serving the last snapshot they have; a process with none serves default parameters and logs no exposures, so it is invisible in the analysis.
Config propagation (p99 < 60 s to 50 k processes). Starts, ramps and stops reach every process within a minute. Starts and ramps are scheduled a couple of minutes ahead so the whole fleet flips at the same instant; stops apply at once.
Result freshness (daily by 06:00 UTC · guardrails < 15 min behind). Decision metrics change slowly and are read once a day. Guardrails exist to catch a broken variant, so they run on a separate streaming path that is minutes behind.
Statistical trustworthiness (real false positive rate ≈ stated α). This is the requirement that drives the hardest trade-off. Teams want answers fast, look every morning and read 100 metrics, and each of those habits multiplies false positives. The platform holds the real rate near 5 % for every team at once, and checks sample ratios on every experiment.
Scale (500 experiments · 300 M users · 50 B events/day). 100 M daily users, 2 M assignment checks a second at peak, and ~5 B exposures plus ~45 B outcome events a day.
Auditability and privacy (every change versioned · deletion within 30 days). Who changed what and when, with the snapshot version stamped on every exposure, so any result can be reproduced. Per-user tables use pseudonymous ids and are purged on account deletion; the results page shows only aggregates.

Back-of-envelope estimates

CPU for assignment, fleet-wide: ~2 cores. A check is two hashes and a few comparisons against an in-memory snapshot: ~1 µs. 2 M checks/s × 1 µs = 2 core-seconds per second, so about two cores across every service. A remote assignment service would instead take 2 M RPCs a second and put a network round trip on each of the dozens of checks in a request. Assignment cost is not the problem; distributing the config is.
Config snapshot size: ~1 MB · ~200 KB compressed. 500 running experiments × ~2 KB each (layer, salt, bucket ranges, variant slots, parameters, compiled targeting) = ~1 MB, about 200 KB compressed. ~50 k server processes hold it, so a full push is 50 k × 200 KB = 10 GB; a delta for one changed experiment is ~2 KB, or ~100 MB fleet-wide.
Events per second: ~580 k/s avg · ~1.5 M/s peak. 50 B events/day ÷ 86 400 ≈ 580 k/s on average, and roughly 2.5× that at the evening peak: ~1.5 M/s. About 5 B of them are exposures; the other ~45 B are the product events that already flow through the company pipeline.
Exposures logged per day: ~5 B (vs ~86 B raw checks). 100 M daily users × ~20 experiments each is exposed to × ~2.5 sessions, logging once per user, experiment and session = 5 B/day. Logging every check instead would be ~1 M checks/s average × 86 400 ≈ 86 B/day, seventeen times more, which is why the SDK deduplicates in the process.
Raw event storage: ~10 TB/day · ~0.7 PB/year compressed. 50 B events × ~200 B = 10 TB/day raw, about 2 TB/day in compressed columnar files. A year is ~0.7 PB. Most of this exists already for product analytics; the platform adds the exposures table, ~0.5 TB/day raw.
Daily join volume: ~2 B rows (vs ~900 B). Roll events up to one row per user per day first: ~100 M rows. Join those to each user’s ~20 running experiments: 100 M × 20 = 2 B rows/day. Joining raw events instead would be 45 B × 20 = 900 B rows, 450 times more. Aggregating per user before the join is the whole trick.
Users per arm for a 1 % lift: ~3 M · ~1.5 M with CUPED. n ≈ 16 σ² / δ² per arm at α = 0.05 two-sided and 80 % power. For 5 % conversion σ² = 0.05 × 0.95 = 0.0475, and a 1 % relative lift is δ = 0.0005: 16 × 0.0475 / 0.00000025 ≈ 3.0 M. CUPED with a pre-period correlation of 0.7 removes about half the variance: ~1.5 M.
Users per experiment if none overlap: ~600 k. 300 M users ÷ 500 experiments = 600 k each, about 300 k per arm, which can only detect lifts of ~3 % on a 5 % conversion rate. Most real wins are under 1 %. Experiments have to share users, so the question becomes how they share them without interfering.

Components

Apps (web, mobile) (cached assignments): The iOS, Android and web clients. At session start they fetch this user’s client-side assignments from the edge, about 2 KB, cache them with a TTL and read parameters from that cache synchronously, so no screen waits on the network. They log an exposure from the same call that reads a parameter, where the experience diverges, in control as well as in treatment.
Edge API (assignments · event intake): The front door for app traffic. Serves GET /v1/assignments by running the same SDK over its in-memory snapshot for this user, or this device when logged out, so phones never download the experiment config and unreleased features never ship inside an app. Also takes the apps’ event batches, produces them to the event log, and forwards product requests to the services.
Product services (in-process SDK): Every backend service (search, ranking, checkout, pricing) embeds the SDK. A check hashes the unit id against the in-memory snapshot in about a microsecond, returns the variant’s parameters and queues an exposure, deduplicated per (unit, experiment) for 30 minutes inside the process. With no snapshot at all, the SDK returns default parameters and logs nothing.
Config snapshots (CDN · immutable versions): Compiled, signed, immutable snapshot files (~1 MB, ~200 KB compressed) and deltas between consecutive versions (~2 KB for one changed experiment), cached on a CDN in front of object storage. SDKs long-poll a tiny latest-version pointer and fetch only what changed, so a change reaches 50 k processes as ~100 MB of deltas rather than 10 GB of snapshots. Every SDK also keeps the last good snapshot on local disk for restarts.
Guardrail monitor (stream · 5-min windows): A streaming job over exposures and guardrail events (crashes, errors, latency, checkout failures), both keyed by unit id. Keeps each user’s exposures from the last 24 h in keyed state, ~32 GB across the cluster, counts guardrail outcomes per variant in 5-minute windows and runs an always-valid sequential test, so looking 288 times a day does not manufacture alarms. On a clear, large harm it stops the variant through the experiment service; it can never ship or restart anything.
Event log (Kafka · keyed by unit id): The company’s existing log carries ~45 B outcome events a day; the platform adds an exposures topic of ~5 B a day. Both are keyed by unit id, so a user’s exposures and events land on the same partition, which lets the guardrail monitor join them without a shuffle. Retained for days, so a broken consumer can rewind instead of losing data.
Experiment console (design · ramp · results): Where product teams design an experiment, run the power calculation, ramp, stop and read results. It enforces the plan: a primary metric, guardrails and a horizon must be chosen before start, and the decision badge of a fixed-horizon test stays “not final” until that date however green the numbers look.
Experiment service (validate · allocate · audit): Owns every experiment’s lifecycle (draft, running, stopped, ended) and every change to it. Validates definitions (each parameter belongs to exactly one layer, metrics exist, guardrails attached), allocates layer buckets in a transaction so two experiments in a layer can never share one, and compiles and publishes a new snapshot version on each change. Refuses changes that would move users between variants and offers a new iteration with a fresh salt instead.
Config store (Postgres · versioned): Experiments and their iterations, layers with their 10,000 buckets and each bucket’s owner, the metric catalog, and an append-only history of config versions with actor, reason and activation time. Megabytes of live state, strongly consistent because bucket ownership is a mutual-exclusion invariant. The pipeline reads the allocation history from here to know the expected split on every day.
Event warehouse (Parquet · by day): All exposures and outcome events in columnar files partitioned by event date and hour: ~10 TB a day raw, ~2 TB compressed. Also holds the pipeline’s derived tables: user-day rollups, kept 90 days so every experiment’s pre-period is available for CUPED, first exposures per experiment, and per-user running totals.
Stats engine (CIs · CUPED · SRM): Turns sums into decisions. Runs the sample ratio check first and replaces the scorecard with a warning if it fails, then computes CUPED-adjusted differences, relative effects with delta-method intervals, fixed-horizon or sequential significance per the experiment’s plan, and false discovery control across secondary metrics. Also answers power calculations at design time from stored baselines.
Results store (per-variant sums): Sufficient statistics per (experiment, iteration, variant, metric, day, slice): n, Σx, Σx² and the CUPED and ratio cross-terms, plus per-metric baselines for power calculations. About 150 k rows a day before slices and ~3 M with them, small enough that any interval is recomputed on request. Never holds per-user data.
Metrics pipeline (Spark · daily): Runs after each day’s partitions close. Rolls 45 B outcome events into one row per user per day, shared by all 500 experiments, joins those rows to first exposures, keeps per-user running totals from first exposure, and reduces them to per-variant sufficient statistics. Everything after it is arithmetic on a few hundred thousand rows; the run takes about two hours.

User flows

A user gets their variants and is exposed. The hot path. Assignment is computed locally wherever it is needed, in microseconds, and the only thing that leaves the process is an exposure log, written asynchronously at the moment the user actually sees the change.
1. The app starts a session and fetches this user’s client-side assignments from the edge. One small request per session, cached on the device for about an hour and read synchronously by the UI, so no screen waits on it. The response holds only the experiments this user is eligible for, about 2 KB, never the full config, so unreleased features do not ship inside the app. A first launch with nothing cached shows defaults and logs no exposures until assignments arrive.
2. The edge evaluates every eligible experiment for the user from its in-memory snapshot. For each layer: bucket = hash(layer salt, user id) mod 10,000, then the experiment that owns that bucket, its targeting against the context (country, platform, app version), then variant = hash(experiment salt and iteration, user id) mod 1,000 against the weights. About a microsecond per experiment and no call to anything. The services run the same function and get the same answer, which is the whole consistency story.
3. The user opens checkout; reading the parameter logs an exposure, which goes up in the app’s next event batch. Exposure is logged where the experience diverges, not at session start: most users who fetched assignments never reach checkout, and counting them would dilute the effect. Control logs from the same call at the same point, because it reached the place where it would have seen the redesign. The app sends one exposure per experiment per session.
4. The checkout request reaches the pricing service, which checks a server-side experiment in-process. The SDK reads its in-memory snapshot, so the check costs about a microsecond and the request waits on nothing. It returns parameters rather than a variant name, so the code reads params.show_fees and never branches on the word “treatment”; a stopped variant is then just different parameters, with no deploy.
5. The service sends the exposure, and later the purchase, to the event log off the request path. The SDK buffers exposures and flushes every second, and a process-local cache drops repeats of (unit, experiment) for 30 minutes, which turns ~86 B raw checks a day into ~5 B exposures. The purchase is an ordinary product event the service already emits. Both are keyed by unit id, so a user’s records share a partition.
6. Exposures and product events are written to the warehouse, partitioned by day. Hourly Parquet files by event date: ~50 B rows a day, ~2 TB compressed. The platform owns only the exposures table; the outcome events are the company’s existing analytics events, which is why metrics are defined over those tables instead of being logged by the SDK.
Launching an experiment. The write path. A definition becomes buckets in a layer and a new snapshot version, and every process starts the experiment at the same instant without anyone calling the platform on the request path.
1. The owner designs the experiment and asks how many users and days it needs. The power calculation uses the metric’s recent mean and variance for this population, stored as baselines, and the variance reduction CUPED usually achieves on it. A 1 % lift on 5 % conversion needs ~3 M users per arm, ~1.5 M with CUPED. The duration is rounded up to whole weeks so every weekday counts equally.
2. The experiment service validates the definition and stores it as a draft. Validation is where the platform’s rules live: every parameter must belong to the experiment’s layer, metrics must exist in the catalog, the standard guardrails (crash rate, error rate, latency, revenue) are attached automatically, and the plan (primary metric, horizon, fixed or sequential) is required. An experiment the platform cannot analyse never starts.
3. On start, the service takes free buckets in the layer and records a new config version, in one transaction. 5 % of the layer is 500 of its 10,000 buckets. The update only takes buckets with no owner, inside a transaction, so two experiments in one layer can never share a bucket: mutual exclusion is enforced by the database, not by teams talking to each other. The iteration gets a fresh random salt.
4. The service compiles a snapshot and publishes it with a delta from the previous version. The snapshot holds, per layer, bucket ranges and their owners, and per experiment its salt, variant slots, parameters and targeting compiled to a small predicate: ~1 MB, ~200 KB compressed. It is signed and immutable. The delta for one changed experiment is ~2 KB, so pushing it to 50 k processes moves ~100 MB instead of 10 GB.
5. Every SDK picks up the new version and swaps it in, ready for the scheduled start. SDKs long-poll the latest pointer, fetch the delta, verify the signature, build the new structure beside the old one and swap a pointer: p99 under 60 s. The experiment carries an active_from two minutes ahead, so every process starts it at the same instant instead of users flickering in and out while the fleet updates. A process that cannot fetch keeps its current snapshot.
6. Exposures start to arrive, and the guardrail monitor begins watching the new experiment. The monitor needs no setup: it learns of the experiment from the snapshot and its first exposures. In the first hours it checks the live sample ratio as well as the guardrails, because a mismatch this early almost always means an exposure-logging bug, and it is cheapest to find before two weeks of traffic are spent.
Daily results for 500 experiments. The scale case. Fifty billion events a day become a few hundred thousand sums, because events are aggregated per user once and shared by every experiment, and everything after that is arithmetic.
1. After the day’s partitions close, the pipeline loads the running experiments and the metric definitions. Metrics are defined once in the catalog, as SQL over the event tables, owned by a team and versioned, so “checkout conversion” means the same thing in every experiment. The allocation history of each experiment, every version’s traffic and weights, is what the sample ratio check compares against.
2. One pass rolls the day’s 45 B outcome events into one row per active user. ~100 M rows with a column per catalog measure. Every experiment reads this table instead of the raw events, so the firehose is scanned once a day rather than 500 times. Kept for 90 days, it also supplies each user’s 14 pre-experiment days for CUPED.
3. First exposures are updated for every experiment and user. One row per (experiment iteration, unit) with the variant and the first exposure time. A unit seen in two variants of one experiment cannot happen with correct hashing, so it is counted per SDK version, alerted above 0.01 % and excluded. Known bots are removed here, in the same way for both arms.
4. The day’s user rows are joined to first exposures and added to each user’s running totals. ~100 M active users × ~20 experiments each is ~2 B joined rows, against ~900 B if raw events were joined to experiments. Only days from first exposure count, and on that first day only events after the exposure time, taken from raw events for just the newly exposed users. Totals are additive, so the job never rescans an experiment’s history.
5. Per-user totals are reduced to a few sums per variant and written to the results store. For each (experiment, variant, metric, slice): n, Σx and Σx², plus the pre-period covariate’s Σc, Σc² and Σxc for CUPED; ratio metrics add their numerator, denominator and cross-product sums for the delta method. That is ~150 k rows a day before slices. Everything after this step is arithmetic on small numbers.
6. The stats engine checks the sample ratio first, then computes effects and intervals. The sample ratio check is a chi-squared test of exposed users against the configured split; below p = 0.0005 the scorecard is replaced by a warning. Then the CUPED-adjusted difference with a Welch interval, the relative effect with a delta-method interval, significance per the plan, and Benjamini-Hochberg across secondary metrics. All from sums, in milliseconds.
7. The owner opens the results page and sees where the experiment stands against its plan. Primary metric first, then guardrails, then secondary metrics marked exploratory. On day 9 of a 14-day fixed-horizon plan the primary metric’s badge says not final, however green it is, and the page plots the effect by day since exposure so a fading novelty effect is visible. The job takes about two hours and results are ready by 06:00 UTC.
A variant raises crashes: automatic stop. The failure path. A daily pipeline is far too slow when a treatment breaks the app, so a streaming monitor watches guardrails with a test that stays valid when checked every five minutes, and a stop reaches every process in under a minute.
1. Android users in an image-cache experiment’s treatment start crashing, and the crash reports arrive with their exposures. Crash reports are uploaded on the next launch and keyed by unit id like everything else, so a user’s crash and their exposures land on the same partition of the event log. The monitor reads only this small subset of events (crashes, errors, latency, checkout failures), not the whole firehose.
2. The guardrail monitor attributes each crash to the experiments the user was exposed to, and counts per variant. Keyed state holds each user’s exposures from the last 24 h: ~2 B (user, experiment) pairs, ~32 GB across the cluster in RocksDB. Crashes before a user’s first exposure are not attributed to the experiment. This is the ad click aggregator’s machinery again (keyed state, event-time windows, checkpoints) in 5-minute windows.
3. Every five minutes it runs a sequential test on each guardrail, for each variant against control. An always-valid test, so 288 looks a day do not produce false alarms the way a repeated t-test would. An automatic stop needs two things: the test crossing its boundary at α = 0.001, and an interval showing a large harm, such as a crash rate entirely above +10 %. With 50 k exposed users per arm, a crash rate doubling from 0.2 % to 0.4 % has a z of almost 6.
4. The monitor asks the experiment service to stop the harmful variant. Stopping sets the variant’s traffic to zero: its users get the default parameters, and its buckets stay owned by the experiment so they are not handed to another one while still carrying this one’s effects. The monitor can only stop; it never restarts, ramps or ships anything.
5. The service commits the change and publishes it on the urgent path, with no scheduled delay. Starts and ramps are scheduled two minutes ahead; stops apply immediately, because every minute is more crashes. The delta is a few hundred bytes. If the experiment service itself is down, an on-call tool writes the same stop as a signed override file beside the snapshots, which SDKs apply on top of their current version.
6. SDKs swap in the new version, and the apps refresh their assignments. Fleet-wide p99 under 60 s. Apps cache assignments for an hour, so the edge returns the current snapshot version in a header on every response, and an app holding a version older than a stop refetches at once. A stop only works for code that reads the parameter at runtime, one more reason experiments are parameters rather than compile-time branches.
7. The owner is paged with the evidence, and a fixed version must start as a new iteration. The page shows the metric, the interval and where the harm concentrates: Android 12 on app version 9.14 here. Restarting reuses nothing: a new iteration gets a new salt, because the users who crashed carry that experience, and the old salt would put exactly those users back into treatment.
Ramping from 5 % to 50 % without moving anyone. Changing a running experiment. Widening the traffic must add users without moving any who are already in, and a change that would move them is refused rather than quietly applied.
1. After a clean first day, the owner ramps the experiment from 5 % to 50 % of the layer. A ramp changes how much of the layer the experiment owns, never the split between its variants, which stays 50/50 at every step. The plan says 5 %, 20 %, 50 %, and each step is allowed only after the guardrails were clean at the one before.
2. The service extends the experiment’s bucket range in the layer, taking only free buckets. Buckets 3,500 to 7,999 join the 3,000 to 3,499 it already owns. If any of them belongs to another experiment the ramp fails: you cannot ramp into someone else’s traffic. The service prefers buckets that have been free longest, so lingering effects from their previous owner have faded.
3. The new version reaches every process and activates at the same instant. Scheduled two minutes ahead, so no user flickers in and out of the experiment while the fleet updates. Exposure volume jumps about tenfold at the activation time, which the guardrail monitor expects because it reads the same snapshot.
4. Users already in keep their variant, and newly included users split evenly. Inclusion comes from the layer hash and the variant from a second hash with the experiment’s own salt, which the ramp does not touch. A user in bucket 3,217 was in treatment before and still is; a user in bucket 5,040 is new and is placed by the same variant hash. The ramp is monotonic, like a percentage rollout of a flag.
5. The owner then asks to change the split to 20/80, and the service refuses to apply it in place. Weights map to variant slots: control holds slots 0 to 499, and 20/80 would shrink it to 0 to 199, moving the users in 200 to 499 into treatment after a week of control. Pooling days with different splits can also reverse the sign of an effect (Simpson’s paradox), because the arms’ mix of early and late users diverges.
6. A new iteration starts with a fresh salt, and its data is analysed separately. Every user is re-randomised once, deliberately, and the new iteration’s analysis starts at its own first exposures. The first iteration’s results stay exactly as they were. Re-randomising also spreads any carry-over from the first iteration evenly across the new arms.
7. The pipeline reads the allocation history, so each day’s expected split is known. Within an iteration the split never changes, so the sample ratio check compares against 50/50 on every day despite the ramp, and pooling days is valid: both arms ramped together, so their mix of early and late users is the same. Users exposed later have less time in the experiment, but equally in both arms.

Deep dives

Assignment without a network call

Why not have every app and service call an assignment service, and store each user’s variant so it never changes?

Assignment sits on the hottest path in the company: 2 M checks a second, dozens per request, on every page and every backend call. A remote assignment service would add a network round trip, about a millisecond and much more at p99, to each of those checks, make every product page depend on the platform being up, and need a table of ~6 B rows (300 M users × ~20 experiments) that grows with every launch. The benefit it offers, an answer that never changes, is exactly what the product needs: a user who flips between variants sees a broken product and pollutes both arms.

The observation that removes the trade-off: a stable answer does not need stored state. If the variant is a pure function of the unit id, the experiment’s salt and a small config that every process holds, then every service, device and restart computes the same answer independently. The only thing left to distribute is the config, which changes a few thousand times a day rather than 2 M times a second.

Deterministic hash in an in-process SDK, evaluated from a pushed snapshot chosen
Central assignment service with a stored variant per user rejected
Random assignment remembered in a cookie or on the device rejected
Ship the full snapshot to every app and evaluate on the device situational: internal or low-stakes apps whose experiment list is not sensitive; consumer apps get only their own ~2 KB from the edge

The answer: Every check is a pure function. The SDK hashes (layer salt, unit id) with a fast, well-mixed hash such as MurmurHash3 into one of 10,000 buckets, finds the experiment that owns that bucket in the layer, evaluates its targeting against attributes the caller passed, then hashes (experiment salt and iteration, unit id) into 1,000 variant slots against the weights. The config is a compiled, signed, versioned snapshot of about 1 MB that every process holds in memory and refreshes from a CDN with ~2 KB deltas, p99 under 60 s, and starts are scheduled a couple of minutes ahead so every process flips at the same instant. With no snapshot at all, the SDK returns default parameters and logs no exposure, so a failure falls back to the control experience and stays out of the analysis. Apps get their assignments from the edge, which runs the same SDK for the user and returns only that user’s variants. Every SDK passes a shared conformance suite of (salt, id) → bucket vectors, because two languages that hash a string differently would silently split one user in two.

A user browses logged out, then logs in halfway through the session. Which variant do they get?

It depends on the experiment’s unit, which the owner must choose. A user-unit experiment cannot be evaluated without a user id, so logged-out traffic gets the defaults and logs no exposure; after login the user is assigned by user id and keeps that variant on every device. A device-unit experiment, such as the signup page, keeps hashing the device id after login, so the experience does not change mid-session. What you must never do is switch the hash input at login within one experiment, because that moves users between arms.

Two SDKs in different languages disagree for 0.1 % of users. How would you find out?

From the data, before anyone reports it. A unit seen in two variants of the same experiment is impossible under correct hashing, so the pipeline counts those per experiment and per SDK version and alerts above a tiny rate. The second symptom is a sample ratio mismatch that appears on one platform only. Prevention is a conformance suite of salts and ids with their expected buckets, including non-ASCII ids, that every SDK runs in CI.

The experiment service is down for an hour. What breaks?

Nothing a user can see. SDKs keep serving their current snapshot, and new processes start from the copy on local disk or from the last version on the CDN, neither of which needs the service. What you lose is change: no launches, no ramps and, most importantly, no stops through the normal path. So stopping has a break-glass route: an on-call tool writes a signed override file next to the snapshots, and SDKs apply it on top of their current version.

Why 10,000 buckets in a layer and not 100?

Granularity and room. With 100 buckets the smallest allocation is 1 % of the layer, and a layer holds at most 100 experiments; 10,000 gives 0.01 % steps for cautious first ramps and space to place experiments without fragmenting the layer. It costs nothing, because the snapshot stores ranges of buckets rather than lists, so a layer is a few dozen ranges whatever the count.

How do you know the assignment is actually random?

Run A/A experiments continuously: two identical variants analysed exactly like real ones, through the same SDK, logging and pipeline. Across many A/A tests the p-values should be uniform, with about 5 % below 0.05, and every one should pass the sample ratio check. If not, something in hashing, exposure logging or the statistics is wrong, and you learn it before a real decision depends on it.

Overlapping experiments: layers and holdouts

Five hundred experiments share 300 M users. Does each get its own slice of users or do they overlap, and how do you stop them interfering?

Traffic is the scarce resource. If every experiment had an exclusive slice, each would get 300 M ÷ 500 = 600 k users, about 300 k per arm, which detects lifts of around 3 % on a 5 % conversion rate. Most real wins are under 1 % and need about 3 M users per arm, so experiments must share users. Sharing can interfere, though: one team tests a blue button background while another tests blue button text, and the users who get both see an invisible button.

Google’s 2010 KDD paper on its overlapping experiment infrastructure (Tang, Agarwal, O’Brien and Meyer) starts from the observation that interference comes from experiments changing the same parameters. Partition the parameters into layers, make experiments within a layer mutually exclusive, and let experiments in different layers overlap freely. A user is in at most one experiment per layer, and in one experiment in each of many layers at once.

An exclusive slice of users for every experiment rejected
Full overlap, every experiment hashed independently situational: a small company with a few dozen experiments on clearly separate surfaces
Layers: exclusive within a layer, orthogonal across layers, domains for holdouts chosen
Full factorial design with the interaction analysed situational: two specific experiments known to interact, where the combination is the decision

The answer: Parameters are registered in exactly one layer (checkout_ui, search_ranking, pricing, notifications…), and validation rejects an experiment that sets a parameter outside its own layer. Each layer has its own salt and 10,000 buckets; an experiment owns bucket ranges assigned by the experiment service in a transaction, so ownership never overlaps, and splits its users into variants with a second hash using its own salt. Before any layer, the SDK checks the domains: a global holdout of 2 % of users per product area per quarter, hashed with its own salt, gets every layer’s defaults and no experiments. Because layer salts are independent, experiments in different layers are crossed evenly, each arm of one containing a proportional mix of the other’s arms, so main effects stay unbiased unless the two genuinely interact. The platform measures the pairs that could: same surface, different layers, overlapping in time.

How would you detect that two experiments in different layers interact?

For each pair running at the same time on the same surface, the pipeline already has users in all four combinations, so it estimates the 2 × 2 interaction term per metric and flags pairs where it is both significant and large. With hundreds of pairs, apply false discovery control or you will chase noise. True interactions turn out to be rare; when one is real, move the two experiments into one layer or run them as a factorial design.

The search ranking layer is full: every bucket is owned. A team wants to start today.

Then they queue, which is the right answer for parameters that genuinely collide. Make the queue shorter: end experiments that reached their horizon, size experiments with the power calculation instead of taking 50 % by habit, and split the layer if it holds parameters that do not really interact, since the ranking model and snippet length can be two layers. The console shows each layer’s occupancy and when buckets will be released, so teams can plan.

Buckets released by one experiment go to the next. Are the new experiment’s users contaminated by what they saw before?

They carry the previous experience, but evenly. The new experiment splits users with its own salt, independent of the old split, so both new arms contain the same mix of old control and old treatment users, and the comparison stays unbiased. A strong lingering effect still adds noise, so the service prefers buckets that have been free longest, and a restarted experiment always gets a new salt rather than reusing the old split.

Every experiment this quarter shipped a 1 % win on conversion. Did conversion rise 30 %?

Almost certainly not, and the global holdout is how you find out. Shipped wins are overestimates because only the ones that came out significant were shipped, the winner’s curse; novelty fades; and some launches improve the same thing. The holdout, 2 % of users who got none of the quarter’s launches, is compared with everyone else at the end of the quarter, and with 6 M held-out users it detects a cumulative change of about half a percent. It costs those users the improvements for a quarter, which is why it is small and time-boxed.

A team needs to change parameters in two layers within one experiment.

Then those parameters interact by definition, and the layering is wrong for them. In order of preference: move the parameters into one layer; or run the experiment in its own domain, a slice of traffic with its own layering; or redesign the layers. Letting one experiment span two layers would break the exclusivity guarantee for every other experiment in both.

Logging exposure, not assignment

Why not log every user at the moment they are bucketed, and analyse everyone assigned?

Assignment happens far earlier and far more often than anyone sees a change. An app fetches assignments for every client-side experiment at session start, and a service checks experiments on requests that never reach the redesigned page. If only 10 % of assigned users ever reach checkout, analysing all of them divides the measured effect by ten while the other 90 % still add their usual noise, so the experiment needs roughly ten times the traffic to see the same real effect. Worse, readers take the diluted number as the size of the effect.

The fix is to record exposure where the experience diverges and analyse only exposed users; Kohavi, Tang and Xu’s book on trustworthy online experiments calls this triggering. The trap is symmetry: control must log at the same point, where the user would have seen the treatment, or the arms stop being comparable. The test for both is the sample ratio: if exposed counts are not in the configured split, the logging is broken somewhere.

Log at assignment, analyse everyone assigned rejected
The getter logs exposure at the divergence point, in every arm chosen
Log exposure when the treatment UI renders rejected
Stamp every outcome event with the user’s current assignments situational: a small subset of events, such as crashes and errors for a guardrail counter

The answer: The SDK’s get() is the only way to read an experiment’s parameters, and it logs the exposure, so the log happens where the code reads the value, in every arm. Teams are told to call it at the divergence point (rendering the checkout page, calling the ranker), and a lint rule flags calls at request entry. Exposures carry unit, experiment, iteration, variant, snapshot version and time; they are deduplicated per (unit, experiment) for 30 minutes in each process and per session on devices, and go to the exposures topic keyed by unit id: ~5 B a day instead of ~86 B raw checks. The analysis counts a user from their first exposure. The sample ratio check runs on exposure counts every day, a chi-squared test against the configured split, and below p = 0.0005 it hides the results behind a warning.

The treatment page is 300 ms slower, so more treatment users leave before the exposure fires. What happens?

Exactly the bias the sample ratio check exists to catch: fewer treatment exposures than the split says, and the treatment users who remain are the patient ones, which flatters the treatment. Logging at the getter, before rendering, mostly removes it because the decision is logged before the slow part. If the check still fails, the result is not trustworthy; the remedy is to fix the logging point and rerun, not to re-weight the arms.

The split is 1,000,000 against 990,000. A 1 % gap. Does it matter?

Yes, a great deal. Against an expected 995,000 each, the chi-squared statistic is about 50, a p-value around 10⁻¹², so this is not chance. Something is losing or adding users in one arm, and whatever it is probably correlates with the outcome: bot filtering that reacts to the treatment, a redirect that drops events, a client version that fails to log. A 1 % imbalance looks small to a person and is enormous to the test.

A prefetch evaluates the checkout experiment for users who never open checkout.

Then exposure is over-counted equally in both arms, which brings back some dilution but no bias. The fix is an SDK call that evaluates without logging, for prefetching and caching, plus an explicit log call when the prefetched page is actually shown. Keeping the default get() logging and the silent call the exception keeps the common case right.

How do you make sure control logs at the same point as treatment?

Make it hard to do anything else: both arms read the same parameter through the same get() call, and the code branches on the returned value afterwards. What breaks symmetry is code that asks “am I in treatment” through a separate path, or reads the experiment only inside the new component, so review guidelines and a lint rule look for exactly that. The sample ratio check is the backstop.

Why put the snapshot version on every exposure?

So every number can be traced to the config that produced it. If a ramp or a stop happened during the day, the pipeline knows which allocation each exposure was made under, which is how it knows the expected split for the sample ratio check and how it spots a process stuck on an old snapshot. It costs four bytes an event.

From 50 B events to a confidence interval

How do you compute 100 metrics for 500 experiments every day without scanning the event firehose 500 times?

The outcome events already exist: 45 B a day of page views, clicks, purchases and errors, landing in the warehouse. A naive pipeline runs one query per experiment that joins its exposures to raw events. That scans ~2 TB of compressed events 500 times a day, and the join is enormous: 45 B events against ~20 experiments per user is ~900 B event-experiment pairs. It also gets the statistics wrong whenever a metric is computed per event instead of per user.

Two observations shape the answer. First, the randomization unit is the user, so the analysis unit must be the user too: one user’s page views are not independent, and variance has to be computed from per-user totals. Second, everything a t-test, CUPED or the delta method needs can be computed from a few sums per variant: the count, the sum, the sum of squares and some cross-products. So aggregate per user once, share that across experiments, and reduce each experiment to a handful of numbers.

One query per experiment over the raw events rejected
Shared user-day rollup, joined to first exposures, reduced to per-variant sums chosen
Streaming computation for every metric situational: guardrails only: a few metrics, short windows, and a decision that cannot wait for tomorrow
Query the warehouse on demand when the page is opened situational: ad hoc investigation of one experiment, not the standard scorecard

The answer: A daily Spark job runs once the day’s partitions close. It rolls the day’s 45 B outcome events into user_day, one row per active user (~100 M) with a column per catalog measure. It updates first_exposure per (experiment iteration, unit) with the variant and time. It joins the day’s user_day rows to first exposures, ~2 B rows, and adds them into per-user running totals since first exposure; on the first day only events after the exposure time count, taken from raw events for just the newly exposed users. Finally it reduces those totals per (experiment, variant, metric, slice) to n, Σx and Σx², the covariate sums Σc, Σc² and Σxc for CUPED, and for ratio metrics the numerator, denominator and cross-product sums for the delta method. It takes about two hours, so results are ready by 06:00 UTC. Percentile metrics such as p95 latency keep a mergeable quantile sketch per variant instead of sums.

Click-through rate is clicks over page views. Why not compute it per page view and run a t-test on that?

Because page views are not independent: one heavy user contributes hundreds of correlated views, so treating views as the sample overstates the sample size and makes intervals too narrow, which means false positives. Randomisation was by user, so compute each user’s clicks and views, take the ratio of sums per variant, and get its variance with the delta method from Σclicks, Σviews, their squares and their cross-product. Same point estimate, honest interval.

Someone finds a bug in the revenue metric after an experiment has run for ten days.

Metric definitions are versioned, so the fix is a new version plus a backfill: recompute that metric’s per-user totals from user_day for the experiment’s days, which is cheap because the rollup is already per user, and rewrite the per-variant sums. The results page shows which definition version produced each number. Never edit the old version in place, or yesterday’s screenshots stop matching anything.

A phone was offline for three days and uploads its events late.

They land in the partitions for their event dates when they arrive, so every run recomputes a trailing window, say the last three days, not only yesterday. Anything later is picked up by a weekly recompute. Recent days are labelled as still settling, the same idea as the ad click aggregator’s provisional days.

How big is the per-user running-total table, and why keep it?

About 4 B (experiment, user) rows across running experiments, each with that experiment’s metric totals and covariates: a few terabytes, rewritten incrementally. It makes the daily job additive, since today’s totals are yesterday’s plus one day, so the job never rescans an experiment’s whole history. It is also where per-user rules apply, such as capping each user’s revenue before it is summed.

A VP wants results every hour, not every day.

For guardrails that already exists, on the streaming path. For decision metrics, hourly results mostly add more looks at the same data, which makes peeking worse, and most metrics have weekly cycles anyway. If the business truly needs it, run the same pipeline incrementally over hourly partitions for a few key metrics, but keep the decision tied to the planned horizon.

Peeking, sequential tests and many metrics

Teams check results every morning and the scorecard has 100 metrics. How does the platform keep the false positive rate anywhere near 5 %?

A p-value below 0.05 means a 5 % chance of a false positive for one test evaluated once. A product manager who checks a 14-day experiment daily and ships on the first green day turns that into about 22 % for a change that does nothing. A scorecard of 100 metrics shows about five green metrics per variant by chance alone. The guardrail monitor is worse still, looking every 5 minutes, 288 times a day. Each is a correct calculation applied the wrong way, and with 500 teams the platform is the one place to fix it for everyone.

Two different remedies fit two different problems. Repeated looks need tests designed for repeated looks: either commit to one look at a fixed horizon, or use sequential methods whose thresholds account for every look. Many metrics need a hierarchy: one primary metric that decides, guardrails that can only block, and everything else treated as exploration, corrected for false discoveries and labelled that way.

Fixed horizon: one decision at the planned end, interim numbers marked not final chosen
Always-valid sequential tests (mSPRT-style confidence sequences) chosen
Group sequential design with a few planned looks (O’Brien-Fleming) situational: high-traffic experiments where stopping a few days early is worth planning for
Report a Bayesian probability to beat control instead of p-values rejected

The answer: Every experiment declares its analysis plan at start: a primary metric, a minimum detectable effect, a horizon from the power calculation rounded up to whole weeks, and fixed-horizon or sequential. Fixed horizon is the default: interim results are visible, the decision badge says not final until the horizon, and shipping early is recorded against the owner. Guardrails always use always-valid sequential tests, because the monitor looks every 5 minutes, and teams that need early stopping on the primary metric opt into a group sequential plan with up to five planned looks, whose boundaries are stored with the plan. For many metrics, the primary metric is tested at α = 0.05, guardrails are tested only for harm, and the rest of the scorecard and every slice by country or platform are corrected with Benjamini-Hochberg at a 5 % false discovery rate and labelled exploratory, to be confirmed by a follow-up experiment. An implausibly large effect gets a Twyman’s law warning, as the Kohavi, Tang and Xu book advises: check the logging before celebrating.

The guardrail monitor stops a variant that was actually fine. What did that cost, and how do you keep it rare?

A stopped variant and an unhappy team, because the experiment loses its treatment users and needs a new iteration. Keep it rare in two ways: the sequential test runs at a low α such as 0.001 however many looks it takes, and an automatic stop also requires a practically large harm, such as a crash-rate interval entirely above +10 %, so a significant but trivial difference only alerts a person. Then measure it: false stops per quarter, from A/A experiments and owner appeals.

Why not just apply Bonferroni across all 100 metrics?

Because it controls the chance of even one false positive among 100, with a threshold near 0.0005 per metric, which makes the scorecard nearly blind and real secondary effects vanish. False discovery control asks a more useful question, what share of the flagged metrics are false, and keeps far more power. Bonferroni is reasonable for a handful of guardrails, where crying wolf is the cost to avoid.

The effect was +4 % in week one and +1 % in week two. Which is real?

Probably neither on its own: that is the shape of a novelty effect, people clicking on something new, and the long-run effect may be lower still. The results page plots the effect by day since first exposure for exactly this reason. Run to the horizon, and for launches where the long run matters, keep a 5 % holdback on the old experience for a month and read it then.

A team runs four variants against one control. Does anything change?

Yes: four comparisons on the primary metric are four chances at a false positive, so the stats engine adjusts for them, with Holm’s method for example, and the power calculation splits traffic five ways rather than two. Every extra variant costs traffic, so the console shows the minimum detectable effect getting worse as variants are added.

How do you prove the platform’s statistics are right?

Run A/A tests constantly through the full production path (same SDK, logging, pipeline and stats engine) and check that their p-values are uniform, that about 5 % fall below 0.05, that sequential tests stop falsely at their stated rate, and that the sample ratio check passes. Replaying past A/A data with simulated effects of known size checks power as well. The platform should be held to the standard it holds product teams to.

Variance reduction with CUPED

A 1 % lift needs 3 M users per arm. How do you get the same answer with fewer users or in less time?

Sample size grows with the variance of the metric: n ≈ 16σ²/δ² per arm at 5 % significance and 80 % power. For a 5 % conversion rate and a 1 % relative lift that is about 3 M users per arm, and engagement metrics such as sessions per user are noisier still because users differ so much from each other. Most of that variance has nothing to do with the experiment: heavy users were heavy before it started and are heavy in both arms.

Microsoft’s 2013 CUPED paper (Deng, Xu, Kohavi and Walker) uses exactly that: subtract the part of each user’s metric that their own pre-experiment behaviour predicts. The covariate was measured before assignment, so it is independent of the variant and the adjustment cannot bias the effect; it only removes noise, by a factor of 1 − ρ², where ρ is the correlation between pre-period and in-experiment values. The paper reported removing around half the variance on some Bing metrics.

CUPED: adjust by the same metric over the 14 days before the experiment chosen
Cap outliers before summing (winsorise) chosen
Run longer situational: the effect is expected to grow over time and the long run is the question
Switch to a more sensitive proxy metric situational: the proxy has been validated against the decision metric across past experiments

The answer: For each metric the pipeline computes the same metric per user over the 14 days before the experiment started, from user_day, which is kept 90 days, and stores Σc, Σc² and Σxc per variant beside n, Σx and Σx². The stats engine estimates θ = cov(x, c) / var(c) pooled across arms and reports the adjusted difference, the difference in means minus θ times the difference in pre-period means, with its variance from the same sums. Users with no pre-period get c = 0, which keeps the estimate unbiased but gives them no reduction. Revenue-like metrics are also capped at a 99.9th percentile fixed from pre-experiment data. At ρ = 0.7, 3 M users per arm become about 1.5 M, and the power calculation already accounts for it, so experiments are planned at the shorter length. The pre-period means double as a check: if they differ significantly between arms, the randomisation or the logging is suspect.

Doesn’t adjusting the metric with another variable let you manufacture a result?

Not if the covariate is fixed before the experiment and chosen before the analysis. The pre-period value cannot be affected by the treatment, so its expected difference between arms is zero, and subtracting θ times that difference changes the variance, not the expected effect. Choosing the covariate after seeing the results would be manipulation, which is why the platform fixes it per metric: the same metric over the 14 pre-experiment days.

Most users in a signup experiment are brand new. Does CUPED help?

Hardly. New users have no pre-period, so their covariate is zero and they get no reduction. Other pre-assignment information still helps a little, such as platform, country or acquisition channel, used as covariates in a regression adjustment, which generalises CUPED. For new-user experiments the honest answer is more traffic or more time.

Why 14 days of pre-period, not 90?

Diminishing returns and drift. The correlation with the in-experiment metric rises quickly over the first week or two and then flattens, while a longer window means more history to keep and more users whose behaviour has changed. Two weeks also covers two full weekly cycles. It is a per-metric setting, so a metric with a monthly rhythm can use more.

How do you apply CUPED to a ratio metric like click-through rate?

Linearise first. With the delta method, a ratio of sums behaves approximately like a mean of a per-user linear combination of numerator and denominator, so the adjustment is built on that linearised value, with the pre-period numerator and denominator as covariates. It all still comes from per-variant sums, a few more cross-products. More bookkeeping, not a different method.

How would you know how much CUPED is saving?

Measure it instead of assuming: the stats engine records the variance with and without the adjustment for every experiment and metric, so the platform reports the reduction per metric. Engagement metrics with stable users gain the most and conversion metrics with many new users the least. Converted into experiment-days saved across 500 experiments, it is usually the single most valuable statistical feature a platform ships.

The theory behind it

A/B test peeking: an interactive playground. Run hundreds of simulated A/B tests, check them every day, and see how peeking turns a 5% false positive rate into 24%, and how a fixed horizon or a sequential boundary fixes it.
Feature flag rollouts: an interactive playground. Roll a feature out to a percentage of 400 users by random choice, by a hash of the user id and by a hash of flag and user, and see flicker, lost access and flags that overlap.
Feature flags and A/B testing: Feature flags for safe releases, how flag evaluation works at scale, gradual rollouts and kill switches, A/B test design with consistent assignment, exposure logging, metrics and guardrails, network effects, and flag hygiene.
OLTP versus OLAP: analytics, columnar storage and data warehouses: Why analytics needs a different store from your application database, row versus columnar storage, data warehouses and lakes, star schemas, pre-aggregation and rollups, and real-time analytics with ClickHouse or Druid.
Batch and stream processing: Stream versus batch, windowing and watermarks, late and out-of-order data, exactly-once counting, approximate algorithms, and the lambda/kappa argument in one paragraph.