A/B test peeking: an interactive simulator of false positives and sequential testing

An A/B test compares a metric between two groups and asks whether the difference is bigger than chance would produce. The p < 0.05 threshold promises a 5% false positive rate, but only if you look once, at a sample size chosen in advance. Peeking, checking the dashboard every day and stopping as soon as the result is significant, quietly turns that 5% into 20% or more.

The playground runs a 30-day test with 1,000 users per arm per day and plots its z score day by day. Run tests where B is no different from A, then ones where it really is 10% better, and compare three stopping rules over 400 tests.

Why peeking inflates false positives

Even with no real difference, the z score wanders as data arrives, and over 30 daily looks it has 30 chances to cross 1.96 by luck. In the playground, stopping at the first significant day declares a false winner in 24% of tests where B and A are identical, against 3.8% for a test evaluated once on day 30. When B really is 10% better, peeking does find it more often and sooner, but it stops on lucky upswings, so the lift it reports averages 20%, twice the truth. Teams that peek ship changes that do nothing and overestimate the ones that work.

Doing it properly

• Fix the sample size first. Detecting a 10% lift on a 5% conversion rate with 80% power needs about 31,000 users per arm. Run until then, and decide once.
• Or use a sequential test, designed to be checked repeatedly. A boundary shaped like O’Brien-Fleming’s demands overwhelming evidence early, here a z of 6.8 on day 3, and about 2.15 at the end. It keeps false positives at 3% while stopping real winners around day 21 instead of 30. Always-valid p-values and Bayesian methods with proper priors are other options.
• Watch guardrail metrics for harm separately: stopping a test because something broke is fine; declaring victory early is not.
• Beware many metrics and many variants: twenty metrics at p < 0.05 give one false win on average. Correct for it or pick one primary metric in advance.

If the goal is to earn as much as possible during the test rather than to measure precisely, a multi-armed bandit shifts traffic towards the better variant as evidence accumulates.

Sample size and the z test in code

from math import sqrt
from statistics import NormalDist

def sample_size(base, lift, alpha=0.05, power=0.8):
    """Users per arm to detect a relative lift with a two-sided test."""
    p1, p2 = base, base * (1 + lift)
    z_a = NormalDist().inv_cdf(1 - alpha / 2)
    z_b = NormalDist().inv_cdf(power)
    var = p1 * (1 - p1) + p2 * (1 - p2)
    return int(((z_a + z_b) ** 2 * var) / (p2 - p1) ** 2) + 1

sample_size(0.05, 0.10)   # about 31,000 users per arm: decide the length before starting

def z_score(conv_a, n_a, conv_b, n_b):
    pool = (conv_a + conv_b) / (n_a + n_b)
    return (conv_b / n_b - conv_a / n_a) / sqrt(pool * (1 - pool) * (1 / n_a + 1 / n_b))

In a system design interview

For an experimentation platform, the statistics belong in the design: “Users are bucketed by hashing their id with the experiment id, so assignment is stable and independent across experiments. Each experiment declares a primary metric and a minimum detectable effect, which fixes its sample size. The dashboard shows sequential confidence intervals, so looking early is safe, and guardrail metrics can stop a test that hurts latency or revenue.” Expect the follow-ups: novelty effects, network effects between users in different arms, and how metrics are computed at scale.

Read the guide to experiments and feature flags or bandits and exploration, see canary deployments, which use a similar comparison to catch bad releases, or watch a strong candidate design Netflix.

Questions people ask