Multi-armed bandits vs A/B tests: an interactive Thompson sampling simulator

An A/B test splits traffic evenly until it ends, so the worse options keep getting their full share while you learn. A multi-armed bandit learns and earns at the same time: it shows the option that looks best most of the time, keeps exploring the others a little, and shifts traffic as evidence accumulates. It suits decisions where the reward during the test matters more than a precise measurement: headlines, thumbnails, recommendations.

The playground serves three headlines 6,000 times. Compare an even split with epsilon-greedy and Thompson sampling, averaged over 20 runs, and click one to see where its traffic went.

Exploration against exploitation

With headlines converting at 4%, 5% and 6%, the even split earns about 302 clicks, 60 fewer than always showing the best one would. Thompson sampling earns about 339, losing 24, because it gives the best headline 68% of impressions on average and the worst one little once the evidence is clear. When one headline is bad, at 1%, the split still gives it a third of the traffic and loses 102 clicks; the bandits lose about 10.

• Epsilon-greedy shows the best-looking option and a random one 10% of the time. Simple, but it keeps spending 10% on options it already knows are bad, and early bad luck can leave it favouring the wrong one: it picked the best headline in 80% of runs against the split’s 95%.
• Thompson sampling keeps a probability distribution of each option’s rate and shows each with the probability that it is the best. Uncertain options get tried; clearly worse ones fade out on their own.
• UCB (upper confidence bound) picks the option with the highest optimistic estimate, mean plus an uncertainty bonus, deterministically.

The trade-off is measurement. A bandit gathers little data on the losing options, so its estimates of how much worse they are stay rough, and with close options it is no better than a test at telling them apart. Use an A/B test when you need to know the effect, for instance before a permanent product change; use a bandit when you just want the most clicks while you find out. Real systems add contextual bandits, which condition on the user and page, and discount old data so they adapt when tastes change.

Thompson sampling in code

import random

class ThompsonSampler:
    """One Beta(clicks + 1, misses + 1) belief per option."""
    def __init__(self, options):
        self.stats = {o: [0, 0] for o in options}      # clicks, impressions

    def choose(self):
        def draw(o):
            clicks, shown = self.stats[o]
            return random.betavariate(clicks + 1, shown - clicks + 1)
        return max(self.stats, key=draw)               # sample a plausible rate for each, take the best

    def record(self, option, clicked):
        self.stats[option][1] += 1
        self.stats[option][0] += int(clicked)

sampler = ThompsonSampler(["headline_a", "headline_b", "headline_c"])
choice = sampler.choose()
sampler.record(choice, clicked=False)

In a system design interview

Bandits come up in feed, ads and notification designs: “For each article, editors supply up to four headlines. The serving layer keeps clicks and impressions per headline in Redis, and a Thompson sampler picks one per request. Counts are updated asynchronously from the click stream, so the sampler works on numbers a few seconds old, which is fine. After a day the winner is locked in.” Expect the follow-ups: delayed rewards such as purchases, position bias in clicks, and how to keep the exploration from hurting new users.

Read the guide to bandits and exploration or experiments and feature flags, see A/B test peeking, or watch a strong candidate design a news feed.

Questions people ask