System design interview question

Design an Ad Serving System

Q: You have 10 M active ads and 100 ms. How do you get from 10 M to the winner, and where does each millisecond go?

Every ad server holds the full index: 10 M compiled ad records (~3 GB) and postings for ~50 targeting terms per ad (~2 GB), refreshed from the change stream. A request becomes ~30 terms (location at several levels, age band, gender, language, device, placement, interest and audience ids), and a conjunction index over the targeting expressions returns ~10 k eligible ads in ~3 ms. Filters on the server’s own memory and the user’s frequency record remove about half: campaign status and schedule, budget state, the pacing coin, caps and user blocks. An in-process pre-ranker on ~20 cheap features ranks the ~5 k by approximate expected value and keeps 200, and the prediction service scores those on GPUs in ~25 ms. The budget is 5 ms gateway, 10 ms user context, 5 ms retrieval, 5 ms filters, 10 ms pre-rank, 35 ms model, 5 ms auction, 5 ms response and 20 ms headroom. It is the retrieve-then-rank shape of a recommender, with one difference that matters: the early stages are hard constraints, and they must be exact.

Q: Why rank by bid times predicted click rate, and why not just charge the winner its bid?

Total value per impression = bid × predicted action rate + quality, in money: a CPM bid counts bid ÷ 1,000, a click bid bid × pCTR, a conversion goal target CPA × pCTR × pCVR. The quality term is a predicted user-experience cost, such as the chance the user hides the ad, converted to money, and it can be negative. Rank by total value, drop anything under the reserve for the placement and country, fill the slots in order, and charge each winner per click (next total − own quality) ÷ own pCTR, capped at its bid and floored at the reserve. Example: A bids $2.00 at 1.5 % with quality +$0.002 (total $0.032), B bids $3.00 at 0.8 % with quality −$0.001 (total $0.023); A wins and pays (0.023 − 0.002) ÷ 0.015 = $1.40 per click. This matches what Google describes publicly for Search (Ad Rank from bid, expected click rate and ad quality, paying what is needed to beat the ad below) and Meta’s published “total value = bid × estimated action rate + ad quality”. Edelman, Ostrovsky and Schwarz (2007) and Varian (2007) showed that generalised second price is not truthful with several slots but has stable outcomes in which advertisers pay what VCG would charge, which is part of why it has held up in practice.

Q: A campaign has $10,000 for today and could spend it all by 9 a.m. How do you spread it without starving it?

Each campaign has a target curve for the day: the cumulative share of the budget that should be spent by each minute, shaped like the eligible traffic it saw over the last 7 days and ending at 23:30 in the advertiser’s time zone. Every 10 s the budget service compares spend with the curve and updates the participation probability p with a proportional-integral controller on the spend rate: the proportional term reacts to the last minute, the integral term closes the accumulated gap over hours. p changes by at most 25 % per step, has a floor of 0.001, and its gains scale with the campaign’s volume, so a campaign spending $50 a day is not swung by one click. Spend in the loop is the expected cost reported at serve time, not billed clicks, so the feedback has no click delay; billed spend corrects the level, not the loop. p rides back to every server in the reply to its next per-second report, and the server flips the coin before scoring. For campaigns whose bid the system already controls, the same controller drives a bid multiplier instead, which buys the cheaper impressions first.

Q: Spend happens on 2,500 servers at once. How do you stop a campaign at its budget without asking a central counter on every auction?

The budget service is sharded by campaign id into 32 in-memory shards, each with a standby, and keeps per campaign committed spend = billed spend from the click aggregator + expected cost of ads served but not yet billed. Every ad server sends each shard one call a second with the spend it recorded since the last call (expected cost at serve: the price for impression billing, price × pCTR per click), ~80 k calls and ~4 M campaign items a second at peak, idempotent by sequence number; the reply carries pacing probabilities and budget states. A campaign is open while more than 5 minutes of its peak spend or 10 click prices remain, whichever is more: servers spend on it freely, and since the service is at most ~2 s behind, open spending cannot reach the budget. Below that it is metered: a server may spend on it only from a slice it leased (about 10 s of its own spend, at least one impression’s charge, expiring after 30 s), slices are granted only out of what remains after the expected cost in flight with at most half of it outstanding, and grants stop once less than one click price is left. The overspend that remains is the gap between expected and billed cost on the last minutes of per-click spend, which is random and trued up within minutes: around 0.1 % for a paced campaign and at most about one click for a small one. Anything above the budget is not billed, and the absorbed dollars are reported daily by budget size.

Q: How do you enforce "at most 3 times a day per user" when one user is served by thousands of servers, sometimes in two regions?

The key is fc:{user_id}, a hash with one field per (campaign, day) holding a count, such as c812:1026 = 2, in an Aerospike or Redis class cluster in each region: ~1 KB per user, TTL 7 days from the last write, with days older than a week dropped on write. The ad server reads the whole record once per request in parallel with the profile, with a 5 ms deadline, checks every capped candidate in memory, and increments the winners’ fields after responding. Users are routed to a home region, so nearly all reads and writes are local; records replicate to the other regions asynchronously within seconds. If the read times out, capped campaigns are excluded for that request and uncapped ones still serve, so a cap is never silently ignored. Counting at serve rather than at render overcounts the ~10 % of served ads that never render, so a cap of 3 a day delivers about 2.7 rendered impressions on average, the conservative side advertisers accept.

Q: The model decides who wins and what they pay. How do you train it on the right data, and how do you test a new model or auction rule before it costs money?

The served log records, per request, the user features exactly as sent to the model, the context, the ad-snapshot version (ad features are 5-minute snapshots, so the version reconstructs them exactly without logging 200 vectors), each winner’s predictions, total value, price and runner-up, 20 sampled losers with their scores, and the pacing probabilities applied. On 1 % of requests the top candidates within a small value margin are shuffled with logged probabilities, which gives unbiased data on ads the model would not have picked. Training joins the log with impressions and clicks by impression id, keeps every click and 10 % of non-clicks, and corrects the predictions for that down-sampling with q = p ÷ (p + (1 − p) ÷ w), the recalibration in Facebook’s 2014 paper “Practical Lessons from Predicting Clicks on Ads at Facebook”; the same paper found that freshness matters, which is the reason for daily retraining with hourly updates. Auction changes such as a new reserve or quality weight are replayed over logged auctions first; model changes are scored offline on normalised entropy and calibration, then canaried on 1 % of traffic. Live experiments split each campaign’s budget between arms in proportion to their traffic, so neither arm can spend the other’s money.

Pick the best of 10 M active ads for 5 M requests a second in under 100 ms, within every campaign’s targeting, budget and frequency caps.

Last updated 2026-10-26. Difficulty: hard. Patterns: auctions, budget-pacing, ranking, inverted-index, low-latency. Reported at Meta, Google, Amazon, Microsoft, TikTok, Pinterest, Snap.

Walk through a strong candidate's answer, turn by turn.

The interviewer asks, the candidate answers and draws, and you press Next. Skip to any stage or key moment, and ask the AI Mentor anything along the way.

Walk through it Take the mock interview

Or sit it in the style of Meta · Google · Amazon · Microsoft, run as that company’s round is reported to go.

The short answer

How to design Ad Serving System in a system design interview, in five steps:

Scope it. Fill the ad slots of a page or feed fetch, match each campaign’s targeting, rank by expected value and charge a fair price, spend daily budgets smoothly, frequency caps per user, advertiser changes apply within seconds, record impressions and clicks for billing, log every decision for learning.
Set the targets. Latency: p99 < 100 ms server side; Scale: 5 M requests/s peak · 10 M active ads; Budget accuracy: overspend < 2 % of daily budget; Delivery: ≥ 95 % of budget spent when traffic allows.
Do the numbers. Ad requests per day: ~2.5 M/s avg · ~216 B/day; Ad servers at peak: ~2,500; Candidates per request: 10 M → ~10 k → ~5 k → 200 → 1–3.
Draw the design. Advertiser API, Campaign store, Index builder, Change stream, Ad gateway, Ad server, Budget & pacing, Click aggregator, Frequency store.
Go deep where it is hard.
- The serving funnel and its 100 ms → In-memory targeting index on every ad server, cheap filters, in-process pre-rank to 200, then the GPU model and the auction.
- Ranking and pricing: the auction → Generalised second price on total value, with a quality term and reserve prices.
- Pacing a budget through the day → Probabilistic throttling: a participation probability per campaign, set every 10 s by a controller against a traffic-shaped target curve.
- Not overspending across 2,500 servers → Spend reported every second while a campaign is far from its limit, leased slices drawn from what remains in its last minutes, trued up by billed spend.
- Frequency caps across regions → One record per user in a regional in-memory key-value store, a count per campaign per day, incremented at serve, replicated asynchronously.
- Logging for training and safe changes → Log every decision with the features sent to the model, predictions, prices and propensities, plus a small exploration slice; replay offline, then run budget-split experiments.

Functional requirements

Fill the ad slots of a page or feed fetch. One request per fetch with 1 to 3 slots and their positions. It returns an ad per slot with a signed tracking token, or nothing for a slot where no ad clears the reserve price: an empty slot is a valid answer, a slow page is not.
Match each campaign’s targeting. Location, age band, gender, language, interests, device, placement, custom audiences (user lists the advertiser uploaded) and exclusions. Targeting is a hard filter: an ad outside it is never shown, however well it would score.
Rank by expected value and charge a fair price. Advertisers bid per impression, per click, or set a target cost per conversion that the system turns into a bid for each request. Rank by bid × predicted action rate plus a quality term; each winner pays the least it needed to keep its place, never more than its bid.
Spend daily budgets smoothly. A $10,000 daily budget is spread over the day in proportion to the traffic the campaign can reach, not spent by 9 a.m. Overspend stays under 2 %, and nothing above the budget is billed.
Frequency caps per user. Per campaign, for example at most 3 impressions per user per day or 10 per week. Approximate is acceptable: a user served from two regions in the same second may see one extra.
Advertiser changes apply within seconds. Pause, bid change, budget change, new targeting and newly approved ads reach every ad server in under 10 s. A paused campaign that keeps spending is a refund and a support ticket.
Record impressions and clicks for billing. Every served ad carries an impression id and its price inside a signed token; the render and click beacons carry it back. Exact counting, deduplication and invoicing happen downstream in the click aggregator.
Log every decision for learning. What was shown, with its predictions, price, the runner-up and a sample of the losers, so the click model trains on what was actually served and a new model or auction rule can be evaluated on past traffic.
Out of scope. Exact counting and invoicing (see Design an Ad Click Aggregator), invalid-traffic detection beyond tagging, the advertiser UI, creative review, conversion attribution, the click model’s internals, and selling our slots to outside bidders through an exchange.

Non-functional requirements

Latency (p99 < 100 ms server side). Split before anything is built: 5 ms gateway, 10 ms user context, 5 ms retrieval, 5 ms filters, 10 ms pre-rank, 35 ms model, 5 ms auction, 5 ms response, 20 ms headroom. The host page renders without the ad if we miss, so a timeout is lost revenue, not an error.
Scale (5 M requests/s peak · 10 M active ads). About 1 M campaigns from 500 k advertisers, shown to 1 B daily users. The fleet is sized by the work per request (candidates filtered and scored), not by the number of ads.
Budget accuracy (overspend < 2 % of daily budget). The requirement that drives the hardest tradeoff in the design: spend is decided on 2,500 servers at once, and a central counter checked on every auction would put a network hop and a hot key on the request path. Spend above the budget is absorbed, so overspend is our money.
Delivery (≥ 95 % of budget spent when traffic allows). The other side of pacing. A controller that never overspends because it throttles too hard under-delivers, and advertisers notice that just as fast.
Freshness of advertiser changes (p99 < 10 s to every ad server). Pauses and bid changes are the ones advertisers watch happen. Newly approved ads go live within a minute.
Availability (99.99 % of requests answered in time). "No ad" counts as an answer. Every dependency has a fallback that keeps the rules: a slow model falls back to the pre-ranker’s scores, a slow frequency store excludes capped campaigns.
Charging correctness (charge ≤ bid · ≤ budget · traceable). Every charge is computed by one auction, carried in a signed token and billed once downstream. The served log must be able to explain any price to an advertiser who asks.
Privacy (consented signals only). Targeting uses only signals the user consented to, sensitive categories are not targetable, custom audiences are matched on hashed ids, and a deleted user’s profile and frequency record are gone within a day.

Back-of-envelope estimates

Ad requests per day: ~2.5 M/s avg · ~216 B/day. Peak is 5 M/s and the daily average about half of it: 2.5 M/s × 86,400 ≈ 216 B requests/day. At about one ad per request that is ~200 B ads served a day, and ~2 B clicks at a 1 % click rate.
Ad servers at peak: ~2,500. About 10 ms of CPU per request: ~1 ms retrieval, ~1 ms filtering 10 k candidates, ~5 ms pre-ranking 5 k at ~1 µs each, the rest auction and response. A 32-core server does ~3,200/s flat out and ~2,000/s at 60 % utilisation: 5 M ÷ 2,000 ≈ 2,500 servers, before N+1 headroom per region.
Candidates per request: 10 M → ~10 k → ~5 k → 200 → 1–3. The targeting index matches ~10 k ads for a typical user in a large market. Pacing throttles skip ~30 %, and caps, exhausted budgets and policy remove ~25 % of the rest: 10 k × 0.7 × 0.75 ≈ 5 k. A light in-process pre-ranker keeps 200 for the model, and the auction fills 1 to 3 slots.
Model scores and GPUs: ~1 B scores/s · ~10,000 GPUs. 5 M requests/s × 200 candidates = 1 B scores/s. At ~100 k scores a second per GPU with batching, that is ~10,000 GPUs at peak. The model is the most expensive box on the diagram, and the 200 is its dial: every candidate added raises its cost linearly.
Ad index per server: ~5 GB. 10 M ads × ~300 B of serving data (ids, bid type and amount, quality prior, creative reference, caps) = 3 GB, plus postings: 10 M ads × ~50 targeting terms × 4 B = 2 GB. ~5 GB fits in memory on every ad server, so retrieval never leaves the process.
Frequency store: ~1 TB per region · 5 M reads/s. 1 B daily users × ~1 KB (up to ~50 capped campaigns seen in the last 7 days, each with a count per day) = 1 TB per region, 3 TB with three replicas, held in memory. One read per request (5 M/s at peak) and ~3 M/s increments, since ~60 % of served ads have a cap.
Served log: ~216 TB/day · ~1.5 PB kept. 216 B requests × ~1 KB (context, the user features sent to the model, winners with predictions and prices, the runner-up, 20 sampled losers, model and snapshot versions) ≈ 216 TB/day. Seven days in full is ~1.5 PB; longer training windows keep every click and 10 % of non-clicks. This is the biggest stream in the system and the model’s only trustworthy training set.
Margin before a campaign is metered: ~$70 · 0.7 % of a $10,000 budget. A $10,000 daily budget averages $0.116/s; at the evening peak about twice that, $0.23/s. Every server reports its spend each second, so the budget service is at most ~2 s behind: ~$0.46 unseen. Metering the campaign once less than 5 minutes of peak spend remains, $0.23 × 300 ≈ $70, leaves a margin 150 times the blind spot. Spending freely before that cannot overspend; only the last $70 needs leased slices.

Components

Advertiser API (campaigns · bids · budgets): Where advertisers and their tools create and edit campaigns, ads, targeting, bids, caps and budgets. Validates each change, writes it to the campaign store with a version, and returns as soon as it commits. It never calls ad servers: everything downstream follows from the change log.
Campaign store (relational · by advertiser): Source of truth for advertisers, campaigns, ads, targeting, budgets and review status, sharded by advertiser id in a MySQL or Spanner class store. Normalised for editing, and never read on the serving path. Every committed change lands in its change log, which the index builder tails.
Index builder (CDC → compiled records): Tails the campaign store’s change log and compiles campaign, ad, targeting and review state into one flat serving record per ad (~300 B) and one per campaign (status, schedule, budget). Publishes them to the change stream, and writes a full snapshot every hour (~5 GB) with the stream offset it matches, which is how a new ad server boots.
Change stream (Kafka · compacted): Compiled serving records keyed by ad id, and campaign records keyed by campaign id, with a version on each. Every ad server tails all of it (a few thousand records a second) and the budget service tails the campaign part. Compaction keeps only the latest record per key, so replaying from the hourly snapshot’s offset takes seconds.
App / web page (feed and page ad slots): A feed fetch or page with ad slots. Asks for ads for the fetch’s slots with their positions, renders the creative, sends an impression beacon once the ad is on screen, and sends clicks through the tracker. It decides nothing about the ad: the signed token it echoes back is the whole contract.
Ad gateway (identity · geo · deadline): Resolves the user or device id, derives location from IP, reads the consent state, and attaches a 100 ms deadline that every downstream call inherits. Routes to the user’s home region and, within it, to an ad server chosen by hashing the user id, so one user’s concurrent requests meet on one server. Any server can take any request if that one is down.
Ad server (index in memory · auction): The decision. Holds the full targeting index and every ad’s serving record in memory (~5 GB), kept current from the change stream. Per request it retrieves ~10 k ads by targeting, filters by budget state, pacing coin and caps to ~5 k, pre-ranks in process to 200, calls the prediction service and runs the auction. It records spend locally and reports it every second, leasing slices only for campaigns near their limit, so no auction waits on a central counter.
Budget & pacing (spend reports · slices · pacing): Owns each campaign’s spend for the day, sharded by campaign id into 32 in-memory shards with a standby each. Every ad server reports its spend to each shard once a second, and the reply carries pacing probabilities and budget states, so pacing needs no second channel. While a campaign has minutes of spend left, servers spend on it freely; in its last few minutes it is metered, and servers may spend only from slices leased out of what remains. Billed spend from the click aggregator replaces the servers’ estimates.
Click aggregator (dedupe · IVT · billing): The Ad Click Aggregator design: deduplicates impressions and clicks by id, tags invalid traffic, counts exactly for billing and reconciles nightly. Here it has one more output: billed spend per campaign every ~10 s, which the budget service uses to replace its estimates. Invalid-traffic credits flow back the same way and give budget back.
Frequency store (KV · TTL · per region): Per-user cap counters in an Aerospike or Redis class in-memory store in each region: one record per user with a count per (campaign, day) for the last 7 days, ~1 KB. Read once per request with a 5 ms deadline and incremented when an ad is served. Replicated between regions asynchronously, which is why caps are approximate.
Prediction service (GPU · pCTR / pCVR): The click and conversion models as a black box: takes the user’s features, the context and 200 ad ids, and returns calibrated pCTR per slot position and pCVR for conversion bids. Runs on GPUs in the same cell as the ad servers with dynamic batching. Holds ad features itself as a versioned snapshot refreshed every 5 minutes, so a request carries 200 ids instead of 200 feature vectors.
Event log (Kafka · served + events): Two topics. ad.served from the ad servers, one ~1 KB record per request; ad.events from the tracker, impressions and clicks keyed by campaign id as the aggregator expects. Both carry the impression id, the join key for billing, training and every dispute.
Event tracker (impression · click beacons): The impression and click endpoints. Verifies the HMAC-signed token minted at serve time (impression id, ad, campaign, price, billing type, expiry), appends the event to the log and, for a click, issues the 302 to the landing page. No lookups, so an event costs microseconds.
Feature store (user + ad features): Online store keyed by user and by ad. The user half is the targeting profile (age band, locations, interests, audience memberships) and the model’s user features, read by the ad server once per request. The ad half (click rates by placement, quality scores, embeddings) is published as 5-minute snapshots that the prediction service loads whole.
Training & calibration (daily + hourly): Joins the served log with impressions and clicks by impression id, retrains the click and conversion models daily with hourly updates, and fits a calibration layer per segment so predicted rates match observed ones. Writes batch features, replays logged auctions to evaluate new models and auction rules, and ships nothing until a canary passes.

User flows

One ad request in 100 ms. The main read path. Ten million ads narrow to ~10 k by targeting, ~5 k after budget, pacing and caps, 200 for the model and 1 to 3 winners, each stage inside its own slice of the 100 ms.
1. The client asks for ads for the two ad slots in the next feed page. Ads are requested per fetch, not per slot, so one auction ranks every candidate once, fills the slots in order and never shows a campaign twice on a page. Positions matter: an ad at position 4 is clicked more than one at 11, so the model predicts per position.
2. The gateway resolves the user and location, attaches a 100 ms deadline, and routes to an ad server in the user’s home region. Location comes from IP and device signals, consent from the user’s settings, and both become targeting terms. The home region keeps the frequency record local, and hashing the user id to one server inside it lets that server catch a user’s concurrent requests. Every server holds the full index, so any of them can take over.
3. The ad server reads the user’s profile and frequency record in parallel. The profile supplies targeting terms (age band, interests, audience ids) and the user features the model needs; the frequency record holds a count per campaign per day for the last 7 days. Each is one key, ~2 ms typical with a 5 ms deadline. A missing frequency record means nothing is capped yet; a timed-out one means capped campaigns are excluded for this request.
4. The in-memory targeting index returns about 10,000 eligible ads, and filters drop the ones that cannot be shown now. The request becomes ~30 terms and the index returns ads whose targeting expression is satisfied in ~3 ms. Then cheap checks on the server’s own memory: campaign active and in schedule, budget open (or a slice held if the campaign is metered), the pacing coin (enter with probability p), caps not reached, advertiser not hidden by this user. About 5,000 survive, and none of the GPU time goes to an ad that could never be shown.
5. A light pre-ranker keeps 200, and the prediction service scores them. The pre-ranker is a small model on ~20 features the server already holds (the ad’s click rate by placement, its bid, a user-ad embedding dot product), ~1 µs per candidate. It ranks by approximate expected value, so a high bid with a modest click rate is not thrown away. The prediction service returns calibrated pCTR per position and pCVR for conversion bids in ~25 ms.
6. The auction ranks by total value, applies the reserve and prices each winner. Total value per impression = bid × predicted action rate + quality, in money: a $2.00 click bid at 1.5 % is $0.030 per impression before quality. The top ads that clear the reserve fill the slots in order, and each pays the least that keeps its place: (next ad’s total − its own quality) ÷ its own pCTR per click, never above its bid. The price is fixed now and signed into the token, so billing never recomputes it.
7. The ad server records each winner’s spend, counts the impressions against caps, and returns the ads with signed tokens. Spend is recorded locally as the expected charge, taken from the campaign’s slice if it is metered: the price for an ad billed per impression, price × pCTR for one billed per click. The cap counter is incremented at serve time, not at render, so a request a second later already sees it. The token carries impression id, ad, campaign, price, billing type and an expiry, HMAC-signed.
8. The served log record is written asynchronously. One record per request: the user features sent to the model, the context, each winner with its predictions, total value and price, the runner-up’s total, 20 sampled losers, the pacing probabilities applied, and the model and ad-snapshot versions. About 1 KB, never on the latency path. It explains every price later and is the only training set that matches what the model saw.
Impression and click: from screen to bill. The write path. Two beacons carry the signed token back, the click aggregator turns them into billable counts, and the same events correct the budget and become training labels.
1. The client renders the ad and sends the impression beacon once it is on screen. Impression billing counts rendered ads, not served ones: about one in ten served ads never renders because the user scrolled past or closed the app. The beacon only echoes the token, so the client cannot change the price or the campaign.
2. The tracker verifies the token and appends the event to the log. An HMAC check and an expiry check, no lookups: a forged token, or one older than an hour, is rejected for the cost of one hash. Events are keyed by campaign id, which is how the aggregator partitions its work.
3. The user clicks; the tracker logs the click and redirects to the landing page. The click URL carries the same token, so the click joins its impression by impression id and a double click is the same id twice. The redirect does not wait for the log write to be acknowledged: a person is waiting, and the aggregator design already accepts losing a fraction of a second of clicks on a crash.
4. The click aggregator dedupes, filters invalid traffic and counts billable events. Exactly the Ad Click Aggregator design: dedupe by impression id and event type, tag bots and click farms, minute windows for dashboards, a nightly recount from the raw archive for the invoice. The price comes from the token, so billing never asks the ad server what anything cost.
5. Billed spend flows back to the budget service every ~10 s and replaces the ad servers’ estimates. Ad servers report expected cost at serve time; for a per-click ad that is price × pCTR, right on average and wrong for any single ad. The budget service keeps committed spend = billed so far + expected cost of ads served in the last few minutes that may still be billed, so an estimate lives for minutes, not all day. Invalid-traffic credits arrive the same way and give budget back.
6. Training joins the served log with impressions and clicks by impression id. Rendered and clicked is a positive, rendered and not clicked within an hour is a negative, never rendered is dropped because nobody saw it. Conversions arrive days later and join on the same id. For long training windows non-clicks are down-sampled to 10 %, and the model is recalibrated for it.
7. New models are calibrated and shipped, and the prediction service loads the matching ad-feature snapshot. Daily retraining with hourly updates, because click models decay within days as ads and users change. A calibration layer per segment (placement, country, bid type) makes predicted rates match observed ones. A canary on 1 % of traffic checks calibration and revenue per request before the rollout continues.
A $50,000 campaign paces through a spike. The scale-breaking case for money. One large campaign spends across 2,500 servers, a live event triples its traffic, and the budget must still last until midnight and stop within 2 %, without a central check on any auction.
1. At midnight in the advertiser’s time zone, the budget service loads the day’s budget and builds the target spend curve. The curve is the share of the budget that should be spent by each minute, shaped like the traffic the campaign can reach over the last 7 days, not a straight line: spending evenly by clock time would overpay at 4 a.m. and starve at 8 p.m. It ends at 23:30, so the last half hour is margin.
2. Every 10 s the controller compares spend with the curve and sets the participation probability. p is the chance the campaign enters an eligible auction. A proportional-integral controller on the spend rate: the last minute running 20 % over target pulls p down by about that much, and the integral term closes the accumulated gap over hours instead of in a burst. p moves at most 25 % per step and never drops below 0.001, so the campaign keeps a trickle of data.
3. Ad servers report their spend every second and receive the new probability in the reply. Each server sends one call per budget shard per second with the expected spend it recorded per campaign since the last call: ~80 k calls and ~4 M campaign items a second at peak. The reply carries p for campaigns whose p changed and any budget state changes, so every server sees a new p within about a second. The campaign is open, far from its limit, so no server asks permission to spend on it.
4. On each eligible request the ad server flips a weighted coin before scoring. Throttling chooses which auctions to enter, not what to bid, so the advertiser’s bid and the auction’s price stay honest. The coin is random per request and its probability is logged, which makes the throttle a known propensity for offline evaluation. At p = 0.42 the campaign skips 58 % of the auctions it could enter.
5. At 20:00 a live event triples traffic; spend runs ahead and the controller cuts the probability within a minute. Spend at three times the target rate shows in the first control step, because servers report expected cost at serve time instead of waiting for clicks. p falls from 0.42 to about 0.15 over five steps, limited to 25 % each, so for about a minute spend runs ahead of the curve, a gap the integral term recovers over the rest of the evening. The budget is hours from its limit, so the spike is a pacing problem, not an overspend risk.
6. Late in the evening less than five minutes of peak spend remains, and the campaign switches to metered slices. At ~$350 left (0.7 % of the budget) the replies mark it metered: from now on a server may spend on it only from a slice it leased, about 10 s of its own spend and at least one impression’s charge, expiring after 30 s. Slices come only out of what remains after the expected cost already in flight, with at most half of it outstanding at once, so the slices themselves can never overspend. A server without a slice skips the campaign and asks for one in its next call.
7. The budget runs out: grants stop, the campaign leaves every auction within ~30 s, and clicks over the limit are not billed. Grants stop once less than one click price remains, and a metered campaign cannot be spent on without a slice, so exhaustion needs no broadcast: the last slices drain or expire within 30 s. Clicks on ads already on screen keep arriving for about a minute, and per-click ads can draw more clicks than expected; the aggregator bills up to the budget and records the rest as absorbed overspend, around 0.1 % for a paced campaign, reported daily.
An ad server dies holding budget slices. The failure path. A crashed server takes its last second of unreported spend and its unspent slices with it; the design must neither lose the money nor let it be spent twice.
1. An ad server crashes with up to a second of unreported spend and slices for a few metered campaigns. In its last second it spent on ~1,500 campaigns without reporting it, and it held slices for the handful of campaigns near their limit that it was serving. Neither can be recovered from the server, so the design treats a server’s memory as disposable and recovers both from elsewhere.
2. For open campaigns the missing second is covered by the margin; metered slices expire after 30 s and count as spent. An open campaign has at least five minutes of peak spend left, and one server’s lost second is a sliver of one second of the fleet’s spend, so nothing can be overspent before billing catches up. A metered campaign’s expired slices are counted as fully spent until billing says otherwise: the safe direction, which can only under-deliver for a few minutes.
3. Impressions and clicks on ads it served still arrive and are billed. Everything billing needs is in the signed tokens on users’ screens: impression id, campaign, price. The dead server is not on the billing path at all, which is the point of signing the price into the token instead of looking it up.
4. Billed spend replaces the estimate and releases what the expired slices did not use. Within a few minutes the billed totals cover the dead server’s real spend, and committed spend drops back to the truth. A crash costs a few minutes of slightly conservative leasing, not money.
5. A replacement boots from the hourly snapshot and replays the change stream from the snapshot’s offset. Loading ~5 GB from object storage takes about a minute, and replaying at most an hour of compacted records a few seconds more. The server reports ready only when its applied position is within 5 s of the stream head, so it never serves a paused campaign from a stale index.
6. It joins the gateway’s pool and gets the pacing table, budget states and the slices it needs. Its first reply from each shard carries the full pacing table and budget states for that shard’s campaigns. A metered campaign it has no slice for is skipped until a slice arrives with the next call, so a cold server is slightly less competitive for a second instead of slow. The gateway ramps its traffic over 30 s.
An advertiser pauses a campaign. The background path. A change committed in the campaign store must reach 2,500 in-memory indexes and the budget service in under 10 s, and a server that boots later must still see it.
1. The advertiser pauses campaign 812; the campaign store commits it with a new version. The API returns as soon as the row commits. It does not call ad servers: 2,500 synchronous calls would make every edit as slow and as fragile as the slowest server.
2. The index builder reads the change from the campaign store’s log. Change data capture from the database log rather than a second write from the API, so a change that committed is never missed and one that rolled back never appears.
3. It publishes a campaign record to the change stream. Status lives on one campaign record that every one of its ads points at, so pausing a campaign with 3,000 ads is one record, not 3,000. Targeting and creative changes recompile the ad records instead.
4. Every ad server applies it to its in-memory index within seconds. Each server tails the whole stream and swaps a record in place only if its version is newer, so a duplicate or out-of-order record is harmless. From the next request the campaign fails the status check. Each server reports the lag from commit_ts to applied; the fleet p99 is the freshness SLO, and a server minutes behind is pulled from serving.
5. The budget service marks the campaign stopped in its next replies. Belt and braces: every server hears from each budget shard once a second, so even a server that missed the stream record stops spending on the campaign within a second or two, and a metered campaign gets no more slices. A budget increase travels the same way and raises the remaining budget on the next control step.
6. The index builder’s next hourly snapshot includes the pause. A server that boots later loads the snapshot and replays from its offset, so it sees the pause whether the pause came before or after the snapshot was cut. Compaction keeps that replay short.

Deep dives

The serving funnel and its 100 ms

You have 10 M active ads and 100 ms. How do you get from 10 M to the winner, and where does each millisecond go?

Every request must respect hard constraints (targeting, schedule, budget, caps, policy) and then find the ad with the highest expected value among what is left. The expensive part is the model: at ~100 k scores a second per GPU and 5 M requests a second, scoring 10 k candidates per request would take 500,000 GPUs. Scoring nothing and ranking by bid would be cheap and would show ads nobody clicks.

The constraints are cheap and exact, and the model is expensive and approximate, so run the cheap stages first on everything and the expensive one on as few ads as possible. Targeting is a boolean expression per ad, so it is answered by an index, not a scan. Budget, pacing and caps are lookups in the ad server’s own memory and one user record. Only the survivors reach a pre-ranker, and only its top 200 reach the GPUs.

In-memory targeting index on every ad server, cheap filters, in-process pre-rank to 200, then the GPU model and the auction chosen
A separate index tier, sharded by ad, queried by scatter-gather situational: the ads no longer fit in one server’s memory: around 100 M ads, or keyword-targeted search ads with huge vocabularies
Score every targeting match with the full model rejected
Retrieve by embedding similarity only, ignoring explicit targeting situational: as an extra source for campaigns with broad or automatic targeting, with the targeting check still applied to what it returns

The answer: Every ad server holds the full index: 10 M compiled ad records (~3 GB) and postings for ~50 targeting terms per ad (~2 GB), refreshed from the change stream. A request becomes ~30 terms (location at several levels, age band, gender, language, device, placement, interest and audience ids), and a conjunction index over the targeting expressions returns ~10 k eligible ads in ~3 ms. Filters on the server’s own memory and the user’s frequency record remove about half: campaign status and schedule, budget state, the pacing coin, caps and user blocks. An in-process pre-ranker on ~20 cheap features ranks the ~5 k by approximate expected value and keeps 200, and the prediction service scores those on GPUs in ~25 ms. The budget is 5 ms gateway, 10 ms user context, 5 ms retrieval, 5 ms filters, 10 ms pre-rank, 35 ms model, 5 ms auction, 5 ms response and 20 ms headroom. It is the retrieve-then-rank shape of a recommender, with one difference that matters: the early stages are hard constraints, and they must be exact.

How do you index targeting with ORs and exclusions, like "US or Canada, age 25 to 44, interested in hiking, not existing customers"?

Compile each ad’s targeting into conjunctions over attributes and index each conjunction under its terms: an ad matches when the request carries one allowed value for every attribute the ad constrains. Exclusions are stored as negative terms checked only for ads that otherwise matched, so they cost nothing for the rest. Yahoo published this approach as “Indexing Boolean Expressions” (VLDB 2009). The expensive part, audience lists with millions of members, is kept on the user side: the profile carries the user’s audience ids, so membership is a term in the request rather than a lookup per ad.

Campaigns targeting all of the US match half of all requests. Do they not swamp the 10 k?

They would if retrieval returned matches in arbitrary order. So postings are sorted by a static score per ad, its recent expected value per impression on the placement, and retrieval stops after the best ~10 k by that score, reading the long broad lists only as far as needed. Narrowly targeted ads have short lists that are read completely, so a well-targeted small advertiser is not crowded out by a broad large one. The static score is refreshed with each compiled record.

What happens if the prediction service is down?

The request goes on with the pre-ranker’s scores, which are calibrated well enough to run the auction at a small revenue cost, and the served log marks it as a fallback so training does not learn from it. Prices are computed from the same scores, so advertisers are charged consistently with how they were ranked. If the pre-ranker is unavailable too, the slot stays empty: an unscored ad has no fair price.

How do you know the pre-ranker is not discarding ads the model would have picked?

Measure it. On 0.01 % of requests, score all ~5 k survivors with the full model after the response has gone, and check how often the winner from the full set was inside the pre-ranker’s 200; the target is above 95 %. A drop usually follows a model release, so the pre-ranker is retrained against each new model’s scores as part of that release.

An advertiser pauses a campaign. How sure are you that all 2,500 servers stopped within 10 s?

Each server reports the lag between a record’s commit time and when it applied it, and the p99 across the fleet is the freshness SLO. A server whose lag grows past a minute takes itself out of the gateway’s pool, because serving from a stale index is worse than serving less. The budget service is the second line: its once-a-second replies mark the campaign stopped, so a server that missed the record stops within a second or two anyway.

Ranking and pricing: the auction

Why rank by bid times predicted click rate, and why not just charge the winner its bid?

Advertisers bid in different units: $20 per thousand impressions, $2 per click, $40 per conversion. To compare them the auction converts each to expected value per impression: a $2 click bid at a 1.5 % predicted click rate is worth $0.030 per impression, a $20 CPM bid $0.020. Ranking by bid alone gives the slot to whoever bids most for an ad nobody clicks: less revenue and a worse feed.

Pricing is a separate decision from ranking. Charging the bid (first price) makes every advertiser guess what the others bid and shade below their value; charging the least needed to win (second price) lets them bid what a click is worth to them. And because prices are computed from predictions, a miscalibrated model misprices every auction it touches.

Generalised second price on total value, with a quality term and reserve prices chosen
First price with bid shading situational: selling slots to outside bidders through an exchange, where the industry moved to first price around 2017 to 2019
VCG: each winner pays the value its presence costs the others situational: pages with several slots of very different value, where truthful bidding matters more than simple explanations
Rank by bid only rejected

The answer: Total value per impression = bid × predicted action rate + quality, in money: a CPM bid counts bid ÷ 1,000, a click bid bid × pCTR, a conversion goal target CPA × pCTR × pCVR. The quality term is a predicted user-experience cost, such as the chance the user hides the ad, converted to money, and it can be negative. Rank by total value, drop anything under the reserve for the placement and country, fill the slots in order, and charge each winner per click (next total − own quality) ÷ own pCTR, capped at its bid and floored at the reserve. Example: A bids $2.00 at 1.5 % with quality +$0.002 (total $0.032), B bids $3.00 at 0.8 % with quality −$0.001 (total $0.023); A wins and pays (0.023 − 0.002) ÷ 0.015 = $1.40 per click. This matches what Google describes publicly for Search (Ad Rank from bid, expected click rate and ad quality, paying what is needed to beat the ad below) and Meta’s published “total value = bid × estimated action rate + ad quality”. Edelman, Ostrovsky and Schwarz (2007) and Varian (2007) showed that generalised second price is not truthful with several slots but has stable outcomes in which advertisers pay what VCG would charge, which is part of why it has held up in practice.

The model overpredicts click rate by 20 % for one advertiser’s new ads. What happens?

They win auctions they should have lost and pay less per click than they should, because the price per click is divided by their pCTR. The clicks do not come, so we collect about a sixth less than the runner-up was worth, and users see a worse ad. That is why calibration, predicted over observed click rate, is monitored per placement, country, bid type and large advertiser with an alert outside 0.95 to 1.05, and why a calibration layer is refit hourly.

An advertiser bid $2.00 and was charged $1.40. They ask why.

Because $1.40 is the least they could have bid and still kept that position, given the ad below them and both ads’ predicted click rates and quality. The served log holds the runner-up’s total, both predictions and the quality terms for that impression, so support can show the arithmetic. "You pay what you needed to beat the next ad, never your whole bid" is also how Google and Meta explain it to their advertisers.

Why did ad exchanges move to first price if second price is truthful?

Publishers started running several auctions in a chain (header bidding), so a second price inside one auction was not the real second price across them, and floors set after seeing bids made the rules easy to bend. First price is simpler to reason about across exchanges, and buyers answered with automated bid shading. Google Ad Manager moved to a unified first-price auction in 2019. Inside our own platform we run the only auction and set the floors, so the argument does not carry over, and second price keeps bids meaningful.

A feed page has two ad slots. One auction or two?

One. Ranking all candidates once and filling the slots in order guarantees a campaign is not shown twice on a page and lets each winner’s price depend on the ad directly below it. Each slot has its own predicted click rate, because position 4 gets clicked more than position 11, so the model returns a pCTR per position. Competitive exclusions, such as two car brands next to each other, are rules applied while filling, not part of the model.

How do you set the reserve price?

Per placement and country, from logged auctions: replay recent traffic with a candidate reserve and measure revenue per thousand requests and fill rate, then confirm with a budget-split experiment. Too high leaves slots empty and lowers revenue; too low lets weak ads in cheaply. A quality floor sits beside it, so an ad users dislike cannot win just by clearing the price.

Pacing a budget through the day

A campaign has $10,000 for today and could spend it all by 9 a.m. How do you spread it without starving it?

A campaign eligible for millions of auctions a day wins enough of them to spend its budget in the first few hours if nothing stops it. That is bad for everyone: the advertiser misses the evening audience, buys early-morning impressions that convert less, and the evening auctions get thin. Stopping at the budget is not pacing; it is running out.

No single server sees the campaign’s spend: it is spread over 2,500 servers, and demand moves minute to minute. So pacing has to be a control loop: a central view of spend against a target, one number that tells every server how much to participate, and a feedback period short enough to catch a spike but long enough not to chase noise.

Probabilistic throttling: a participation probability per campaign, set every 10 s by a controller against a traffic-shaped target curve chosen
Bid modification: scale the bid by a multiplier the controller sets situational: campaigns whose bid the system already sets, such as target cost per conversion or maximise conversions
Even hourly caps: budget ÷ 24 per hour rejected
No pacing: serve until the budget is gone situational: an advertiser explicitly chooses accelerated delivery for a time-critical launch

The answer: Each campaign has a target curve for the day: the cumulative share of the budget that should be spent by each minute, shaped like the eligible traffic it saw over the last 7 days and ending at 23:30 in the advertiser’s time zone. Every 10 s the budget service compares spend with the curve and updates the participation probability p with a proportional-integral controller on the spend rate: the proportional term reacts to the last minute, the integral term closes the accumulated gap over hours. p changes by at most 25 % per step, has a floor of 0.001, and its gains scale with the campaign’s volume, so a campaign spending $50 a day is not swung by one click. Spend in the loop is the expected cost reported at serve time, not billed clicks, so the feedback has no click delay; billed spend corrects the level, not the loop. p rides back to every server in the reply to its next per-second report, and the server flips the coin before scoring. For campaigns whose bid the system already controls, the same controller drives a bid multiplier instead, which buys the cheaper impressions first.

Spend swings 30 % above and below the target every few minutes. What is wrong?

The controller is reacting faster than its measurement settles: too high a gain, or a delay in the loop. Check the delay first: a loop fed by billed clicks acts on spend minutes old and keeps overshooting, which is why it uses expected cost at serve time. Then lower the proportional gain for that campaign’s volume class, smooth the measured rate over a minute instead of 10 s, and keep the per-step limit on p. The metric is the hourly spend-to-curve error across campaigns, and how its spread changes after any controller change.

A new campaign starts at 15:00 with no history. What is its curve, and its first p?

Its curve is the placement’s average traffic shape for the remaining hours, scaled so the budget lands by 23:30, and it is replaced by the campaign’s own traffic after a day. Its first p comes from a forecast: eligible requests per minute from the index’s match counts, times a win rate estimated from similar bids, solved for the target spend rate. Start below the forecast and let the controller raise it, because overspending the first hour is harder to undo than under-spending it.

The campaign is at p = 1 and still spends half its target. What now?

Pacing cannot spend money the auction will not give it: the campaign is losing on bid or relevance, or its targeting is too narrow. The system reports it as limited by bid or by audience, with the estimated delivery at a higher bid, instead of under-delivering silently. Under-delivery is measured only for campaigns that ended the day below p = 1, so pacing faults are not mixed up with bidding ones.

Budgets reset at midnight in each advertiser’s time zone. Is there a spike?

Not a global one: resets are spread over 24 time zones, and each campaign’s curve starts slowly in its own night. What to avoid is every campaign in a zone starting at p = 1, so the first p of the day is yesterday’s p at the same point of the curve. The budget service processes a zone’s resets as one batch at that zone’s midnight.

How do you know pacing is working?

Three numbers a day. The share of campaigns that delivered 95 to 102 % of budget where traffic allowed. The hourly spend-to-curve error. And the absorbed overspend in dollars, by budget size, which for paced campaigns should be around 0.1 % of spend. A shift in any of them after a controller change is a rollback.

Not overspending across 2,500 servers

Spend happens on 2,500 servers at once. How do you stop a campaign at its budget without asking a central counter on every auction?

A central counter is the obvious answer: every win decrements the campaign’s remaining budget atomically. At 5 M requests a second that is 5 M decrements a second on the request path, a network hop each, and a large campaign’s counter becomes a hot key taking tens of thousands of writes a second. When the counter is slow, every auction is slow; when it is down, either nothing serves or everything overspends. Leasing each server a slice of every campaign’s budget fixes the hop but not the scale: 1 M campaigns × 2,500 servers is 2.5 B leases, mostly idle, and a slice cannot be smaller than one impression’s charge.

Two observations make it tractable. Most campaigns are far from their limit most of the day: a paced $10,000 campaign spends $0.23 a second at peak, so a budget service that hears from every server each second is never more than about 50 cents behind on it. And overspend need not be zero, only bounded: the requirement is under 2 %, and anything above the budget is absorbed rather than billed. So exact control is needed only for the last few minutes of each budget, and only there do servers need permission to spend.

Spend reported every second while a campaign is far from its limit, leased slices drawn from what remains in its last minutes, trued up by billed spend chosen
Leased slices for every campaign, all day situational: a few thousand large campaigns or a few dozen servers, where a lease per server and campaign is cheap
Central atomic counter per campaign, decremented on every win rejected
Spend counted downstream from the click stream, with a pause signal situational: as the true-up and safety net under the budget service, never as the only control

The answer: The budget service is sharded by campaign id into 32 in-memory shards, each with a standby, and keeps per campaign committed spend = billed spend from the click aggregator + expected cost of ads served but not yet billed. Every ad server sends each shard one call a second with the spend it recorded since the last call (expected cost at serve: the price for impression billing, price × pCTR per click), ~80 k calls and ~4 M campaign items a second at peak, idempotent by sequence number; the reply carries pacing probabilities and budget states. A campaign is open while more than 5 minutes of its peak spend or 10 click prices remain, whichever is more: servers spend on it freely, and since the service is at most ~2 s behind, open spending cannot reach the budget. Below that it is metered: a server may spend on it only from a slice it leased (about 10 s of its own spend, at least one impression’s charge, expiring after 30 s), slices are granted only out of what remains after the expected cost in flight with at most half of it outstanding, and grants stop once less than one click price is left. The overspend that remains is the gap between expected and billed cost on the last minutes of per-click spend, which is random and trued up within minutes: around 0.1 % for a paced campaign and at most about one click for a small one. Anything above the budget is not billed, and the absorbed dollars are reported daily by budget size.

A $20 daily budget with a $2 click price. How does that work across 2,500 servers?

Ten click prices is $20, so the campaign is metered from its first impression: a server may spend on it only from a slice, and with at most half of what remains outstanding, a handful of servers hold slices at any moment while the rest skip it. That costs nothing, because a $20 campaign wants about ten clicks out of millions of eligible auctions. Grants stop when less than one $2 click remains, so it ends within about one click of its budget, and a click that lands over it is absorbed rather than billed.

Why record expected cost at serve time for per-click ads? The click may never come.

Because waiting for clicks puts a delay of seconds to minutes into both the budget and the pacing loop, and a delayed control loop overshoots. Expected cost, price × pCTR, is unbiased when the model is calibrated and is known the moment the ad is served. Billed spend replaces it within minutes, so the error only lives in the last few minutes’ window. A miscalibrated model shows up here first, as a growing gap between expected and billed spend.

A budget shard is down for 20 s. What happens to its campaigns?

Open campaigns keep serving, because their margin is minutes of spend, and servers buffer their reports and resend them with the same sequence numbers when the standby takes over, which deduplicates them. Metered campaigns on that shard spend only the slices already held, at most about 10 s of each server’s spend, and are then skipped until the shard is back: they fail closed, with a bound. If an outage outlasts 60 s, servers treat the shard’s open campaigns as metered too and stop spending on them, because the margin was sized for seconds of blindness, not minutes.

The advertiser raises today’s budget from $10,000 to $15,000 at 14:00.

The change arrives on the change stream; the budget service raises the remaining budget at once and rescales the rest of the target curve, so the extra $5,000 is spread over the remaining hours instead of spent in the next one. p rises over a few control steps. Lowering a budget below today’s spend stops the campaign immediately: the next replies mark it stopped, and outstanding slices expire within 30 s.

At 10x the traffic, does the budget service hold up?

Spend items grow with served ads, so ~40 M a second, still batched into one call per server per shard per second. Campaigns are independent, so adding shards scales it linearly. The cheaper lever is the report interval: every 2 s halves the work, and a 5-minute margin is still 75 times the blind spot. Leases grow only with the number of campaigns near their limit at once, a small fraction of all campaigns.

Frequency caps across regions

How do you enforce "at most 3 times a day per user" when one user is served by thousands of servers, sometimes in two regions?

A cap is checked on every request and changed on every served ad: 5 M reads and ~3 M increments a second at peak, inside the 10 ms user-context slice. The state is per user and per campaign, 1 B users with up to ~50 capped campaigns each over a 7-day window: ~1 TB per region. It has to come back in one round trip, because one request can have thousands of capped candidates.

Caps are a promise about user experience, not money, so approximate is acceptable as long as the error is small and in a known direction. One extra impression for a user who switched regions within a second is fine; caps that silently stop working when the store is slow are not. That points to one record per user, read locally, written once per served ad, and replicated between regions in the background.

One record per user in a regional in-memory key-value store, a count per campaign per day, incremented at serve, replicated asynchronously chosen
Strongly consistent global counters rejected
Counts kept on the device and sent with the request situational: logged-out web traffic or privacy-restricted contexts with no stable user id
Count impressions from the event stream instead of at serve time situational: caps defined on viewable impressions for brand campaigns, alongside the serve-time count as the fast guard

The answer: The key is fc:{user_id}, a hash with one field per (campaign, day) holding a count, such as c812:1026 = 2, in an Aerospike or Redis class cluster in each region: ~1 KB per user, TTL 7 days from the last write, with days older than a week dropped on write. The ad server reads the whole record once per request in parallel with the profile, with a 5 ms deadline, checks every capped candidate in memory, and increments the winners’ fields after responding. Users are routed to a home region, so nearly all reads and writes are local; records replicate to the other regions asynchronously within seconds. If the read times out, capped campaigns are excluded for that request and uncapped ones still serve, so a cap is never silently ignored. Counting at serve rather than at render overcounts the ~10 % of served ads that never render, so a cap of 3 a day delivers about 2.7 rendered impressions on average, the conservative side advertisers accept.

The feed requests two pages in parallel for the same user. Both read a count of 2 and both show the ad.

That is the known race: two reads before either increment. Within one request it cannot happen, because the slots are filled by one auction and a campaign wins at most once. Across concurrent requests, the gateway hashes the user id to one ad server in the region, and that server keeps a few seconds of "just served" notes per user in memory, which closes the race in practice without a lock in the store.

Why not a Bloom filter or count-min sketch to save memory?

A sketch answers "probably seen" with false positives, and here a false positive silently suppresses an ad for a user who never saw it: lost delivery nobody can debug. A terabyte per region is affordable at this scale. Sketches are the right tool for estimating unique reach across campaigns, not for per-user caps.

How do you size and shard the store?

By a hash of the user id across ~50 nodes per region with 64 GB each, which holds ~3 TB with three replicas and takes tens of thousands of operations a second per node at peak. There are no hot keys, because one person makes a few requests a minute. The thing to watch is record size: an advertiser running hundreds of capped campaigns at one audience grows records, so fields per user are capped at a few hundred with the oldest evicted first.

Why count at serve time and not when the ad renders?

Speed. A user can make the next request a second later, before the render beacon has been processed, and a count from the event stream would let both pass. Counting at serve means the next request already sees it. The price is counting served ads that never rendered, which errs toward fewer impressions; brand campaigns that want caps on viewable impressions get a second check from the stream on top.

A user deletes their account. What happens to the caps?

The record is deleted in every region along with the rest of the user’s profile, and the TTL would remove it within 7 days of the last ad anyway. It holds only campaign ids and counts under a user id, no browsing data, so nothing in it outlives the user.

Logging for training and safe changes

The model decides who wins and what they pay. How do you train it on the right data, and how do you test a new model or auction rule before it costs money?

The click model is trained on the system’s own output: it only sees outcomes for ads it chose to show, in positions it chose, at prices it set. If training rebuilds features later from a warehouse, it learns from numbers it never saw while serving. If it only sees winners, it never learns that an ad it scored low would have done well. And because prices are computed from predictions, a model that ranks well but is miscalibrated still charges the wrong price.

A live A/B test is the final word, but for ads it is expensive and subtly wrong: campaigns in both arms share one budget, so the arm that spends faster takes budget from the other and looks better than it is. So the logs must support offline evaluation first, and live experiments must split budgets between arms.

Log every decision with the features sent to the model, predictions, prices and propensities, plus a small exploration slice; replay offline, then run budget-split experiments chosen
Train on impression and click logs, rebuilding features from the warehouse rejected
Ship changes straight to an A/B test on live traffic situational: the last step for a change that already passed an offline replay, with budgets split between arms

The answer: The served log records, per request, the user features exactly as sent to the model, the context, the ad-snapshot version (ad features are 5-minute snapshots, so the version reconstructs them exactly without logging 200 vectors), each winner’s predictions, total value, price and runner-up, 20 sampled losers with their scores, and the pacing probabilities applied. On 1 % of requests the top candidates within a small value margin are shuffled with logged probabilities, which gives unbiased data on ads the model would not have picked. Training joins the log with impressions and clicks by impression id, keeps every click and 10 % of non-clicks, and corrects the predictions for that down-sampling with q = p ÷ (p + (1 − p) ÷ w), the recalibration in Facebook’s 2014 paper “Practical Lessons from Predicting Clicks on Ads at Facebook”; the same paper found that freshness matters, which is the reason for daily retraining with hourly updates. Auction changes such as a new reserve or quality weight are replayed over logged auctions first; model changes are scored offline on normalised entropy and calibration, then canaried on 1 % of traffic. Live experiments split each campaign’s budget between arms in proportion to their traffic, so neither arm can spend the other’s money.

Why log the user features instead of reading them again at training time?

Because the user’s features change by the minute, and what matters is their value at the moment of the decision. Logging the ~1 KB that was actually sent makes training free of skew by construction. Ad features change too, but they are versioned snapshots shared by every request, so logging the version reconstructs them exactly at a fraction of the size.

Ads at position 11 get fewer clicks than at position 4. Does the model learn that bad ads go low?

It would if position were not an input: ads shown low get fewer clicks because of where they were, and a model without position would learn the ads were worse. So position is a feature in training, and at serving the model predicts for each slot’s position. The exploration slice, which sometimes swaps near-tied ads between slots, supplies the data that separates the position effect from the ad’s quality.

How would you evaluate a 10 % higher reserve price before shipping it?

Replay last week’s logged auctions with the new reserve: candidates, predictions and bids are in the log, and a reserve change does not change which ads were retrieved, so winners, prices and empty slots can be recomputed. That gives revenue per thousand requests and fill rate per placement. It cannot show advertisers raising bids in response, or pacing and budgets behaving differently, which is why a budget-split experiment follows before rollout.

How do you know the model in production is still calibrated?

Compare predicted with observed click rate hourly, per placement, country, bid type and large advertiser, from the served log joined with clicks. A ratio outside 0.95 to 1.05 alerts, because prices depend on it directly. The calibration layer is refit hourly on fresh data, and a sudden shift right after a model release means a rollback rather than a refit.

A new ad has no history. How does it get a fair chance?

Its prediction comes from content and advertiser features, shrunk toward the advertiser’s average click rate until it has impressions of its own, and the exploration slice gives it some auctions where it is close to the top but not at it. A few thousand impressions are enough to learn its rate, after which it competes on its own data. Thompson sampling over the uncertain estimate is the usual tool, and the logged probabilities keep that data usable for training and evaluation.

The theory behind it

Ad auctions: an interactive playground. Bid for an ad slot against three rivals ranked by bid times click-through rate, switch between first-price and second-price rules, and see your profit at every bid.
Ad serving and real-time auctions: How an ad system picks and serves an ad in under 100 ms: targeting and candidate retrieval, click prediction, auctions and pricing, budgets and pacing, frequency caps, impression and click tracking, fraud filtering and billing-grade counting.
Multi-armed bandits and exploration: Learning what works while serving users: the explore-exploit trade-off, epsilon-greedy, upper confidence bounds and Thompson sampling, contextual bandits, bandits versus A/B tests, uses in ranking, notifications, pricing and creative selection, logging for off-policy evaluation, and pitfalls.
Feature stores and serving machine learning models: The infrastructure behind production machine learning: offline and online feature stores, training-serving skew, point-in-time correct training data, real-time features from streams, model serving with latency budgets, batching, shadow and canary deployments, and monitoring drift.