Design an Ad Serving System

• Fill the ad slots of a page or feed fetch: One request per fetch with 1 to 3 slots and their positions. It returns an ad per slot with a signed tracking token, or nothing for a slot where no ad clears the reserve price: an empty slot is a valid answer, a slow page is not.
• Match each campaign’s targeting: Location, age band, gender, language, interests, device, placement, custom audiences (user lists the advertiser uploaded) and exclusions. Targeting is a hard filter: an ad outside it is never shown, however well it would score.
• Rank by expected value and charge a fair price: Advertisers bid per impression, per click, or set a target cost per conversion that the system turns into a bid for each request. Rank by bid × predicted action rate plus a quality term; each winner pays the least it needed to keep its place, never more than its bid.
• Spend daily budgets smoothly: A $10,000 daily budget is spread over the day in proportion to the traffic the campaign can reach, not spent by 9 a.m. Overspend stays under 2 %, and nothing above the budget is billed.
• Frequency caps per user: Per campaign, for example at most 3 impressions per user per day or 10 per week. Approximate is acceptable: a user served from two regions in the same second may see one extra.
• Advertiser changes apply within seconds: Pause, bid change, budget change, new targeting and newly approved ads reach every ad server in under 10 s. A paused campaign that keeps spending is a refund and a support ticket.
• Record impressions and clicks for billing: Every served ad carries an impression id and its price inside a signed token; the render and click beacons carry it back. Exact counting, deduplication and invoicing happen downstream in the click aggregator.
• Log every decision for learning: What was shown, with its predictions, price, the runner-up and a sample of the losers, so the click model trains on what was actually served and a new model or auction rule can be evaluated on past traffic.
• Out of scope: Exact counting and invoicing (see Design an Ad Click Aggregator), invalid-traffic detection beyond tagging, the advertiser UI, creative review, conversion attribution, the click model’s internals, and selling our slots to outside bidders through an exchange.

• Latency (p99 < 100 ms server side): Split before anything is built: 5 ms gateway, 10 ms user context, 5 ms retrieval, 5 ms filters, 10 ms pre-rank, 35 ms model, 5 ms auction, 5 ms response, 20 ms headroom. The host page renders without the ad if we miss, so a timeout is lost revenue, not an error.
• Scale (5 M requests/s peak · 10 M active ads): About 1 M campaigns from 500 k advertisers, shown to 1 B daily users. The fleet is sized by the work per request (candidates filtered and scored), not by the number of ads.
• Budget accuracy (overspend < 2 % of daily budget): The requirement that drives the hardest tradeoff in the design: spend is decided on 2,500 servers at once, and a central counter checked on every auction would put a network hop and a hot key on the request path. Spend above the budget is absorbed, so overspend is our money.
• Delivery (≥ 95 % of budget spent when traffic allows): The other side of pacing. A controller that never overspends because it throttles too hard under-delivers, and advertisers notice that just as fast.
• Freshness of advertiser changes (p99 < 10 s to every ad server): Pauses and bid changes are the ones advertisers watch happen. Newly approved ads go live within a minute.
• Availability (99.99 % of requests answered in time): "No ad" counts as an answer. Every dependency has a fallback that keeps the rules: a slow model falls back to the pre-ranker’s scores, a slow frequency store excludes capped campaigns.
• Charging correctness (charge ≤ bid · ≤ budget · traceable): Every charge is computed by one auction, carried in a signed token and billed once downstream. The served log must be able to explain any price to an advertiser who asks.
• Privacy (consented signals only): Targeting uses only signals the user consented to, sensitive categories are not targetable, custom audiences are matched on hashed ids, and a deleted user’s profile and frequency record are gone within a day.

• Ad requests per day: ~2.5 M/s avg · ~216 B/day. Peak is 5 M/s and the daily average about half of it: 2.5 M/s × 86,400 ≈ 216 B requests/day. At about one ad per request that is ~200 B ads served a day, and ~2 B clicks at a 1 % click rate.
• Ad servers at peak: ~2,500. About 10 ms of CPU per request: ~1 ms retrieval, ~1 ms filtering 10 k candidates, ~5 ms pre-ranking 5 k at ~1 µs each, the rest auction and response. A 32-core server does ~3,200/s flat out and ~2,000/s at 60 % utilisation: 5 M ÷ 2,000 ≈ 2,500 servers, before N+1 headroom per region.
• Candidates per request: 10 M → ~10 k → ~5 k → 200 → 1–3. The targeting index matches ~10 k ads for a typical user in a large market. Pacing throttles skip ~30 %, and caps, exhausted budgets and policy remove ~25 % of the rest: 10 k × 0.7 × 0.75 ≈ 5 k. A light in-process pre-ranker keeps 200 for the model, and the auction fills 1 to 3 slots.
• Model scores and GPUs: ~1 B scores/s · ~10,000 GPUs. 5 M requests/s × 200 candidates = 1 B scores/s. At ~100 k scores a second per GPU with batching, that is ~10,000 GPUs at peak. The model is the most expensive box on the diagram, and the 200 is its dial: every candidate added raises its cost linearly.
• Ad index per server: ~5 GB. 10 M ads × ~300 B of serving data (ids, bid type and amount, quality prior, creative reference, caps) = 3 GB, plus postings: 10 M ads × ~50 targeting terms × 4 B = 2 GB. ~5 GB fits in memory on every ad server, so retrieval never leaves the process.
• Frequency store: ~1 TB per region · 5 M reads/s. 1 B daily users × ~1 KB (up to ~50 capped campaigns seen in the last 7 days, each with a count per day) = 1 TB per region, 3 TB with three replicas, held in memory. One read per request (5 M/s at peak) and ~3 M/s increments, since ~60 % of served ads have a cap.
• Served log: ~216 TB/day · ~1.5 PB kept. 216 B requests × ~1 KB (context, the user features sent to the model, winners with predictions and prices, the runner-up, 20 sampled losers, model and snapshot versions) ≈ 216 TB/day. Seven days in full is ~1.5 PB; longer training windows keep every click and 10 % of non-clicks. This is the biggest stream in the system and the model’s only trustworthy training set.
• Margin before a campaign is metered: ~$70 · 0.7 % of a $10,000 budget. A $10,000 daily budget averages $0.116/s; at the evening peak about twice that, $0.23/s. Every server reports its spend each second, so the budget service is at most ~2 s behind: ~$0.46 unseen. Metering the campaign once less than 5 minutes of peak spend remains, $0.23 × 300 ≈ $70, leaves a margin 150 times the blind spot. Spending freely before that cannot overspend; only the last $70 needs leased slices.

• Advertiser API (campaigns · bids · budgets): Where advertisers and their tools create and edit campaigns, ads, targeting, bids, caps and budgets. Validates each change, writes it to the campaign store with a version, and returns as soon as it commits. It never calls ad servers: everything downstream follows from the change log.
• Campaign store (relational · by advertiser): Source of truth for advertisers, campaigns, ads, targeting, budgets and review status, sharded by advertiser id in a MySQL or Spanner class store. Normalised for editing, and never read on the serving path. Every committed change lands in its change log, which the index builder tails.
• Index builder (CDC → compiled records): Tails the campaign store’s change log and compiles campaign, ad, targeting and review state into one flat serving record per ad (~300 B) and one per campaign (status, schedule, budget). Publishes them to the change stream, and writes a full snapshot every hour (~5 GB) with the stream offset it matches, which is how a new ad server boots.
• Change stream (Kafka · compacted): Compiled serving records keyed by ad id, and campaign records keyed by campaign id, with a version on each. Every ad server tails all of it (a few thousand records a second) and the budget service tails the campaign part. Compaction keeps only the latest record per key, so replaying from the hourly snapshot’s offset takes seconds.
• App / web page (feed and page ad slots): A feed fetch or page with ad slots. Asks for ads for the fetch’s slots with their positions, renders the creative, sends an impression beacon once the ad is on screen, and sends clicks through the tracker. It decides nothing about the ad: the signed token it echoes back is the whole contract.
• Ad gateway (identity · geo · deadline): Resolves the user or device id, derives location from IP, reads the consent state, and attaches a 100 ms deadline that every downstream call inherits. Routes to the user’s home region and, within it, to an ad server chosen by hashing the user id, so one user’s concurrent requests meet on one server. Any server can take any request if that one is down.
• Ad server (index in memory · auction): The decision. Holds the full targeting index and every ad’s serving record in memory (~5 GB), kept current from the change stream. Per request it retrieves ~10 k ads by targeting, filters by budget state, pacing coin and caps to ~5 k, pre-ranks in process to 200, calls the prediction service and runs the auction. It records spend locally and reports it every second, leasing slices only for campaigns near their limit, so no auction waits on a central counter.
• Budget & pacing (spend reports · slices · pacing): Owns each campaign’s spend for the day, sharded by campaign id into 32 in-memory shards with a standby each. Every ad server reports its spend to each shard once a second, and the reply carries pacing probabilities and budget states, so pacing needs no second channel. While a campaign has minutes of spend left, servers spend on it freely; in its last few minutes it is metered, and servers may spend only from slices leased out of what remains. Billed spend from the click aggregator replaces the servers’ estimates.
• Click aggregator (dedupe · IVT · billing): The Ad Click Aggregator design: deduplicates impressions and clicks by id, tags invalid traffic, counts exactly for billing and reconciles nightly. Here it has one more output: billed spend per campaign every ~10 s, which the budget service uses to replace its estimates. Invalid-traffic credits flow back the same way and give budget back.
• Frequency store (KV · TTL · per region): Per-user cap counters in an Aerospike or Redis class in-memory store in each region: one record per user with a count per (campaign, day) for the last 7 days, ~1 KB. Read once per request with a 5 ms deadline and incremented when an ad is served. Replicated between regions asynchronously, which is why caps are approximate.
• Prediction service (GPU · pCTR / pCVR): The click and conversion models as a black box: takes the user’s features, the context and 200 ad ids, and returns calibrated pCTR per slot position and pCVR for conversion bids. Runs on GPUs in the same cell as the ad servers with dynamic batching. Holds ad features itself as a versioned snapshot refreshed every 5 minutes, so a request carries 200 ids instead of 200 feature vectors.
• Event log (Kafka · served + events): Two topics. ad.served from the ad servers, one ~1 KB record per request; ad.events from the tracker, impressions and clicks keyed by campaign id as the aggregator expects. Both carry the impression id, the join key for billing, training and every dispute.
• Event tracker (impression · click beacons): The impression and click endpoints. Verifies the HMAC-signed token minted at serve time (impression id, ad, campaign, price, billing type, expiry), appends the event to the log and, for a click, issues the 302 to the landing page. No lookups, so an event costs microseconds.
• Feature store (user + ad features): Online store keyed by user and by ad. The user half is the targeting profile (age band, locations, interests, audience memberships) and the model’s user features, read by the ad server once per request. The ad half (click rates by placement, quality scores, embeddings) is published as 5-minute snapshots that the prediction service loads whole.
• Training & calibration (daily + hourly): Joins the served log with impressions and clicks by impression id, retrains the click and conversion models daily with hourly updates, and fits a calibration layer per segment so predicted rates match observed ones. Writes batch features, replays logged auctions to evaluate new models and auction rules, and ships nothing until a canary passes.

1. One ad request in 100 ms: The main read path. Ten million ads narrow to ~10 k by targeting, ~5 k after budget, pacing and caps, 200 for the model and 1 to 3 winners, each stage inside its own slice of the 100 ms.
   1. The client asks for ads for the two ad slots in the next feed page.
   2. The gateway resolves the user and location, attaches a 100 ms deadline, and routes to an ad server in the user’s home region.
   3. The ad server reads the user’s profile and frequency record in parallel.
   4. The in-memory targeting index returns about 10,000 eligible ads, and filters drop the ones that cannot be shown now.
   5. A light pre-ranker keeps 200, and the prediction service scores them.
   6. The auction ranks by total value, applies the reserve and prices each winner.
   7. The ad server records each winner’s spend, counts the impressions against caps, and returns the ads with signed tokens.
   8. The served log record is written asynchronously.
2. Impression and click: from screen to bill: The write path. Two beacons carry the signed token back, the click aggregator turns them into billable counts, and the same events correct the budget and become training labels.
   1. The client renders the ad and sends the impression beacon once it is on screen.
   2. The tracker verifies the token and appends the event to the log.
   3. The user clicks; the tracker logs the click and redirects to the landing page.
   4. The click aggregator dedupes, filters invalid traffic and counts billable events.
   5. Billed spend flows back to the budget service every ~10 s and replaces the ad servers’ estimates.
   6. Training joins the served log with impressions and clicks by impression id.
   7. New models are calibrated and shipped, and the prediction service loads the matching ad-feature snapshot.
3. A $50,000 campaign paces through a spike: The scale-breaking case for money. One large campaign spends across 2,500 servers, a live event triples its traffic, and the budget must still last until midnight and stop within 2 %, without a central check on any auction.
   1. At midnight in the advertiser’s time zone, the budget service loads the day’s budget and builds the target spend curve.
   2. Every 10 s the controller compares spend with the curve and sets the participation probability.
   3. Ad servers report their spend every second and receive the new probability in the reply.
   4. On each eligible request the ad server flips a weighted coin before scoring.
   5. At 20:00 a live event triples traffic; spend runs ahead and the controller cuts the probability within a minute.
   6. Late in the evening less than five minutes of peak spend remains, and the campaign switches to metered slices.
   7. The budget runs out: grants stop, the campaign leaves every auction within ~30 s, and clicks over the limit are not billed.
4. An ad server dies holding budget slices: The failure path. A crashed server takes its last second of unreported spend and its unspent slices with it; the design must neither lose the money nor let it be spent twice.
   1. An ad server crashes with up to a second of unreported spend and slices for a few metered campaigns.
   2. For open campaigns the missing second is covered by the margin; metered slices expire after 30 s and count as spent.
   3. Impressions and clicks on ads it served still arrive and are billed.
   4. Billed spend replaces the estimate and releases what the expired slices did not use.
   5. A replacement boots from the hourly snapshot and replays the change stream from the snapshot’s offset.
   6. It joins the gateway’s pool and gets the pacing table, budget states and the slices it needs.
5. An advertiser pauses a campaign: The background path. A change committed in the campaign store must reach 2,500 in-memory indexes and the budget service in under 10 s, and a server that boots later must still see it.
   1. The advertiser pauses campaign 812; the campaign store commits it with a new version.
   2. The index builder reads the change from the campaign store’s log.
   3. It publishes a campaign record to the change stream.
   4. Every ad server applies it to its in-memory index within seconds.
   5. The budget service marks the campaign stopped in its next replies.
   6. The index builder’s next hourly snapshot includes the pause.

Ask: You have 10 M active ads and 100 ms. How do you get from 10 M to the winner, and where does each millisecond go?

Good answers name: In-memory targeting index on every ad server, cheap filters, in-process pre-rank to 200, then the GPU model and the auction, A separate index tier, sharded by ad, queried by scatter-gather, Score every targeting match with the full model, Retrieve by embedding similarity only, ignoring explicit targeting.

Our pick: Every ad server holds the full index: 10 M compiled ad records (~3 GB) and postings for ~50 targeting terms per ad (~2 GB), refreshed from the change stream. A request becomes ~30 terms (location at several levels, age band, gender, language, device, placement, interest and audience ids), and a conjunction index over the targeting expressions returns ~10 k eligible ads in ~3 ms. Filters on the server’s own memory and the user’s frequency record remove about half: campaign status and schedule, budget state, the pacing coin, caps and user blocks. An in-process pre-ranker on ~20 cheap features ranks the ~5 k by approximate expected value and keeps 200, and the prediction service scores those on GPUs in ~25 ms. The budget is 5 ms gateway, 10 ms user context, 5 ms retrieval, 5 ms filters, 10 ms pre-rank, 35 ms model, 5 ms auction, 5 ms response and 20 ms headroom. It is the retrieve-then-rank shape of a recommender, with one difference that matters: the early stages are hard constraints, and they must be exact.

1. How do you index targeting with ORs and exclusions, like "US or Canada, age 25 to 44, interested in hiking, not existing customers"?
   Compile each ad’s targeting into conjunctions over attributes and index each conjunction under its terms: an ad matches when the request carries one allowed value for every attribute the ad constrains. Exclusions are stored as negative terms checked only for ads that otherwise matched, so they cost nothing for the rest. Yahoo published this approach as “Indexing Boolean Expressions” (VLDB 2009). The expensive part, audience lists with millions of members, is kept on the user side: the profile carries the user’s audience ids, so membership is a term in the request rather than a lookup per ad.
2. Campaigns targeting all of the US match half of all requests. Do they not swamp the 10 k?
   They would if retrieval returned matches in arbitrary order. So postings are sorted by a static score per ad, its recent expected value per impression on the placement, and retrieval stops after the best ~10 k by that score, reading the long broad lists only as far as needed. Narrowly targeted ads have short lists that are read completely, so a well-targeted small advertiser is not crowded out by a broad large one. The static score is refreshed with each compiled record.
3. What happens if the prediction service is down?
   The request goes on with the pre-ranker’s scores, which are calibrated well enough to run the auction at a small revenue cost, and the served log marks it as a fallback so training does not learn from it. Prices are computed from the same scores, so advertisers are charged consistently with how they were ranked. If the pre-ranker is unavailable too, the slot stays empty: an unscored ad has no fair price.
4. How do you know the pre-ranker is not discarding ads the model would have picked?
   Measure it. On 0.01 % of requests, score all ~5 k survivors with the full model after the response has gone, and check how often the winner from the full set was inside the pre-ranker’s 200; the target is above 95 %. A drop usually follows a model release, so the pre-ranker is retrained against each new model’s scores as part of that release.
5. An advertiser pauses a campaign. How sure are you that all 2,500 servers stopped within 10 s?
   Each server reports the lag between a record’s commit time and when it applied it, and the p99 across the fleet is the freshness SLO. A server whose lag grows past a minute takes itself out of the gateway’s pool, because serving from a stale index is worse than serving less. The budget service is the second line: its once-a-second replies mark the campaign stopped, so a server that missed the record stops within a second or two anyway.

Ask: Why rank by bid times predicted click rate, and why not just charge the winner its bid?

Good answers name: Generalised second price on total value, with a quality term and reserve prices, First price with bid shading, VCG: each winner pays the value its presence costs the others, Rank by bid only.

Our pick: Total value per impression = bid × predicted action rate + quality, in money: a CPM bid counts bid ÷ 1,000, a click bid bid × pCTR, a conversion goal target CPA × pCTR × pCVR. The quality term is a predicted user-experience cost, such as the chance the user hides the ad, converted to money, and it can be negative. Rank by total value, drop anything under the reserve for the placement and country, fill the slots in order, and charge each winner per click (next total − own quality) ÷ own pCTR, capped at its bid and floored at the reserve. Example: A bids $2.00 at 1.5 % with quality +$0.002 (total $0.032), B bids $3.00 at 0.8 % with quality −$0.001 (total $0.023); A wins and pays (0.023 − 0.002) ÷ 0.015 = $1.40 per click. This matches what Google describes publicly for Search (Ad Rank from bid, expected click rate and ad quality, paying what is needed to beat the ad below) and Meta’s published “total value = bid × estimated action rate + ad quality”. Edelman, Ostrovsky and Schwarz (2007) and Varian (2007) showed that generalised second price is not truthful with several slots but has stable outcomes in which advertisers pay what VCG would charge, which is part of why it has held up in practice.

1. The model overpredicts click rate by 20 % for one advertiser’s new ads. What happens?
   They win auctions they should have lost and pay less per click than they should, because the price per click is divided by their pCTR. The clicks do not come, so we collect about a sixth less than the runner-up was worth, and users see a worse ad. That is why calibration, predicted over observed click rate, is monitored per placement, country, bid type and large advertiser with an alert outside 0.95 to 1.05, and why a calibration layer is refit hourly.
2. An advertiser bid $2.00 and was charged $1.40. They ask why.
   Because $1.40 is the least they could have bid and still kept that position, given the ad below them and both ads’ predicted click rates and quality. The served log holds the runner-up’s total, both predictions and the quality terms for that impression, so support can show the arithmetic. "You pay what you needed to beat the next ad, never your whole bid" is also how Google and Meta explain it to their advertisers.
3. Why did ad exchanges move to first price if second price is truthful?
   Publishers started running several auctions in a chain (header bidding), so a second price inside one auction was not the real second price across them, and floors set after seeing bids made the rules easy to bend. First price is simpler to reason about across exchanges, and buyers answered with automated bid shading. Google Ad Manager moved to a unified first-price auction in 2019. Inside our own platform we run the only auction and set the floors, so the argument does not carry over, and second price keeps bids meaningful.
4. A feed page has two ad slots. One auction or two?
   One. Ranking all candidates once and filling the slots in order guarantees a campaign is not shown twice on a page and lets each winner’s price depend on the ad directly below it. Each slot has its own predicted click rate, because position 4 gets clicked more than position 11, so the model returns a pCTR per position. Competitive exclusions, such as two car brands next to each other, are rules applied while filling, not part of the model.
5. How do you set the reserve price?
   Per placement and country, from logged auctions: replay recent traffic with a candidate reserve and measure revenue per thousand requests and fill rate, then confirm with a budget-split experiment. Too high leaves slots empty and lowers revenue; too low lets weak ads in cheaply. A quality floor sits beside it, so an ad users dislike cannot win just by clearing the price.

Ask: A campaign has $10,000 for today and could spend it all by 9 a.m. How do you spread it without starving it?

Good answers name: Probabilistic throttling: a participation probability per campaign, set every 10 s by a controller against a traffic-shaped target curve, Bid modification: scale the bid by a multiplier the controller sets, Even hourly caps: budget ÷ 24 per hour, No pacing: serve until the budget is gone.

Our pick: Each campaign has a target curve for the day: the cumulative share of the budget that should be spent by each minute, shaped like the eligible traffic it saw over the last 7 days and ending at 23:30 in the advertiser’s time zone. Every 10 s the budget service compares spend with the curve and updates the participation probability p with a proportional-integral controller on the spend rate: the proportional term reacts to the last minute, the integral term closes the accumulated gap over hours. p changes by at most 25 % per step, has a floor of 0.001, and its gains scale with the campaign’s volume, so a campaign spending $50 a day is not swung by one click. Spend in the loop is the expected cost reported at serve time, not billed clicks, so the feedback has no click delay; billed spend corrects the level, not the loop. p rides back to every server in the reply to its next per-second report, and the server flips the coin before scoring. For campaigns whose bid the system already controls, the same controller drives a bid multiplier instead, which buys the cheaper impressions first.

1. Spend swings 30 % above and below the target every few minutes. What is wrong?
   The controller is reacting faster than its measurement settles: too high a gain, or a delay in the loop. Check the delay first: a loop fed by billed clicks acts on spend minutes old and keeps overshooting, which is why it uses expected cost at serve time. Then lower the proportional gain for that campaign’s volume class, smooth the measured rate over a minute instead of 10 s, and keep the per-step limit on p. The metric is the hourly spend-to-curve error across campaigns, and how its spread changes after any controller change.
2. A new campaign starts at 15:00 with no history. What is its curve, and its first p?
   Its curve is the placement’s average traffic shape for the remaining hours, scaled so the budget lands by 23:30, and it is replaced by the campaign’s own traffic after a day. Its first p comes from a forecast: eligible requests per minute from the index’s match counts, times a win rate estimated from similar bids, solved for the target spend rate. Start below the forecast and let the controller raise it, because overspending the first hour is harder to undo than under-spending it.
3. The campaign is at p = 1 and still spends half its target. What now?
   Pacing cannot spend money the auction will not give it: the campaign is losing on bid or relevance, or its targeting is too narrow. The system reports it as limited by bid or by audience, with the estimated delivery at a higher bid, instead of under-delivering silently. Under-delivery is measured only for campaigns that ended the day below p = 1, so pacing faults are not mixed up with bidding ones.
4. Budgets reset at midnight in each advertiser’s time zone. Is there a spike?
   Not a global one: resets are spread over 24 time zones, and each campaign’s curve starts slowly in its own night. What to avoid is every campaign in a zone starting at p = 1, so the first p of the day is yesterday’s p at the same point of the curve. The budget service processes a zone’s resets as one batch at that zone’s midnight.
5. How do you know pacing is working?
   Three numbers a day. The share of campaigns that delivered 95 to 102 % of budget where traffic allowed. The hourly spend-to-curve error. And the absorbed overspend in dollars, by budget size, which for paced campaigns should be around 0.1 % of spend. A shift in any of them after a controller change is a rollback.

Ask: Spend happens on 2,500 servers at once. How do you stop a campaign at its budget without asking a central counter on every auction?

Good answers name: Spend reported every second while a campaign is far from its limit, leased slices drawn from what remains in its last minutes, trued up by billed spend, Leased slices for every campaign, all day, Central atomic counter per campaign, decremented on every win, Spend counted downstream from the click stream, with a pause signal.

Our pick: The budget service is sharded by campaign id into 32 in-memory shards, each with a standby, and keeps per campaign committed spend = billed spend from the click aggregator + expected cost of ads served but not yet billed. Every ad server sends each shard one call a second with the spend it recorded since the last call (expected cost at serve: the price for impression billing, price × pCTR per click), ~80 k calls and ~4 M campaign items a second at peak, idempotent by sequence number; the reply carries pacing probabilities and budget states. A campaign is open while more than 5 minutes of its peak spend or 10 click prices remain, whichever is more: servers spend on it freely, and since the service is at most ~2 s behind, open spending cannot reach the budget. Below that it is metered: a server may spend on it only from a slice it leased (about 10 s of its own spend, at least one impression’s charge, expiring after 30 s), slices are granted only out of what remains after the expected cost in flight with at most half of it outstanding, and grants stop once less than one click price is left. The overspend that remains is the gap between expected and billed cost on the last minutes of per-click spend, which is random and trued up within minutes: around 0.1 % for a paced campaign and at most about one click for a small one. Anything above the budget is not billed, and the absorbed dollars are reported daily by budget size.

1. A $20 daily budget with a $2 click price. How does that work across 2,500 servers?
   Ten click prices is $20, so the campaign is metered from its first impression: a server may spend on it only from a slice, and with at most half of what remains outstanding, a handful of servers hold slices at any moment while the rest skip it. That costs nothing, because a $20 campaign wants about ten clicks out of millions of eligible auctions. Grants stop when less than one $2 click remains, so it ends within about one click of its budget, and a click that lands over it is absorbed rather than billed.
2. Why record expected cost at serve time for per-click ads? The click may never come.
   Because waiting for clicks puts a delay of seconds to minutes into both the budget and the pacing loop, and a delayed control loop overshoots. Expected cost, price × pCTR, is unbiased when the model is calibrated and is known the moment the ad is served. Billed spend replaces it within minutes, so the error only lives in the last few minutes’ window. A miscalibrated model shows up here first, as a growing gap between expected and billed spend.
3. A budget shard is down for 20 s. What happens to its campaigns?
   Open campaigns keep serving, because their margin is minutes of spend, and servers buffer their reports and resend them with the same sequence numbers when the standby takes over, which deduplicates them. Metered campaigns on that shard spend only the slices already held, at most about 10 s of each server’s spend, and are then skipped until the shard is back: they fail closed, with a bound. If an outage outlasts 60 s, servers treat the shard’s open campaigns as metered too and stop spending on them, because the margin was sized for seconds of blindness, not minutes.
4. The advertiser raises today’s budget from $10,000 to $15,000 at 14:00.
   The change arrives on the change stream; the budget service raises the remaining budget at once and rescales the rest of the target curve, so the extra $5,000 is spread over the remaining hours instead of spent in the next one. p rises over a few control steps. Lowering a budget below today’s spend stops the campaign immediately: the next replies mark it stopped, and outstanding slices expire within 30 s.
5. At 10x the traffic, does the budget service hold up?
   Spend items grow with served ads, so ~40 M a second, still batched into one call per server per shard per second. Campaigns are independent, so adding shards scales it linearly. The cheaper lever is the report interval: every 2 s halves the work, and a 5-minute margin is still 75 times the blind spot. Leases grow only with the number of campaigns near their limit at once, a small fraction of all campaigns.

Ask: How do you enforce "at most 3 times a day per user" when one user is served by thousands of servers, sometimes in two regions?

Good answers name: One record per user in a regional in-memory key-value store, a count per campaign per day, incremented at serve, replicated asynchronously, Strongly consistent global counters, Counts kept on the device and sent with the request, Count impressions from the event stream instead of at serve time.

Our pick: The key is fc:{user_id}, a hash with one field per (campaign, day) holding a count, such as c812:1026 = 2, in an Aerospike or Redis class cluster in each region: ~1 KB per user, TTL 7 days from the last write, with days older than a week dropped on write. The ad server reads the whole record once per request in parallel with the profile, with a 5 ms deadline, checks every capped candidate in memory, and increments the winners’ fields after responding. Users are routed to a home region, so nearly all reads and writes are local; records replicate to the other regions asynchronously within seconds. If the read times out, capped campaigns are excluded for that request and uncapped ones still serve, so a cap is never silently ignored. Counting at serve rather than at render overcounts the ~10 % of served ads that never render, so a cap of 3 a day delivers about 2.7 rendered impressions on average, the conservative side advertisers accept.

1. The feed requests two pages in parallel for the same user. Both read a count of 2 and both show the ad.
   That is the known race: two reads before either increment. Within one request it cannot happen, because the slots are filled by one auction and a campaign wins at most once. Across concurrent requests, the gateway hashes the user id to one ad server in the region, and that server keeps a few seconds of "just served" notes per user in memory, which closes the race in practice without a lock in the store.
2. Why not a Bloom filter or count-min sketch to save memory?
   A sketch answers "probably seen" with false positives, and here a false positive silently suppresses an ad for a user who never saw it: lost delivery nobody can debug. A terabyte per region is affordable at this scale. Sketches are the right tool for estimating unique reach across campaigns, not for per-user caps.
3. How do you size and shard the store?
   By a hash of the user id across ~50 nodes per region with 64 GB each, which holds ~3 TB with three replicas and takes tens of thousands of operations a second per node at peak. There are no hot keys, because one person makes a few requests a minute. The thing to watch is record size: an advertiser running hundreds of capped campaigns at one audience grows records, so fields per user are capped at a few hundred with the oldest evicted first.
4. Why count at serve time and not when the ad renders?
   Speed. A user can make the next request a second later, before the render beacon has been processed, and a count from the event stream would let both pass. Counting at serve means the next request already sees it. The price is counting served ads that never rendered, which errs toward fewer impressions; brand campaigns that want caps on viewable impressions get a second check from the stream on top.
5. A user deletes their account. What happens to the caps?
   The record is deleted in every region along with the rest of the user’s profile, and the TTL would remove it within 7 days of the last ad anyway. It holds only campaign ids and counts under a user id, no browsing data, so nothing in it outlives the user.

Ask: The model decides who wins and what they pay. How do you train it on the right data, and how do you test a new model or auction rule before it costs money?

Good answers name: Log every decision with the features sent to the model, predictions, prices and propensities, plus a small exploration slice; replay offline, then run budget-split experiments, Train on impression and click logs, rebuilding features from the warehouse, Ship changes straight to an A/B test on live traffic.

Our pick: The served log records, per request, the user features exactly as sent to the model, the context, the ad-snapshot version (ad features are 5-minute snapshots, so the version reconstructs them exactly without logging 200 vectors), each winner’s predictions, total value, price and runner-up, 20 sampled losers with their scores, and the pacing probabilities applied. On 1 % of requests the top candidates within a small value margin are shuffled with logged probabilities, which gives unbiased data on ads the model would not have picked. Training joins the log with impressions and clicks by impression id, keeps every click and 10 % of non-clicks, and corrects the predictions for that down-sampling with q = p ÷ (p + (1 − p) ÷ w), the recalibration in Facebook’s 2014 paper “Practical Lessons from Predicting Clicks on Ads at Facebook”; the same paper found that freshness matters, which is the reason for daily retraining with hourly updates. Auction changes such as a new reserve or quality weight are replayed over logged auctions first; model changes are scored offline on normalised entropy and calibration, then canaried on 1 % of traffic. Live experiments split each campaign’s budget between arms in proportion to their traffic, so neither arm can spend the other’s money.

1. Why log the user features instead of reading them again at training time?
   Because the user’s features change by the minute, and what matters is their value at the moment of the decision. Logging the ~1 KB that was actually sent makes training free of skew by construction. Ad features change too, but they are versioned snapshots shared by every request, so logging the version reconstructs them exactly at a fraction of the size.
2. Ads at position 11 get fewer clicks than at position 4. Does the model learn that bad ads go low?
   It would if position were not an input: ads shown low get fewer clicks because of where they were, and a model without position would learn the ads were worse. So position is a feature in training, and at serving the model predicts for each slot’s position. The exploration slice, which sometimes swaps near-tied ads between slots, supplies the data that separates the position effect from the ad’s quality.
3. How would you evaluate a 10 % higher reserve price before shipping it?
   Replay last week’s logged auctions with the new reserve: candidates, predictions and bids are in the log, and a reserve change does not change which ads were retrieved, so winners, prices and empty slots can be recomputed. That gives revenue per thousand requests and fill rate per placement. It cannot show advertisers raising bids in response, or pacing and budgets behaving differently, which is why a budget-split experiment follows before rollout.
4. How do you know the model in production is still calibrated?
   Compare predicted with observed click rate hourly, per placement, country, bid type and large advertiser, from the served log joined with clicks. A ratio outside 0.95 to 1.05 alerts, because prices depend on it directly. The calibration layer is refit hourly on fresh data, and a sudden shift right after a model release means a rollback rather than a refit.
5. A new ad has no history. How does it get a fair chance?
   Its prediction comes from content and advertiser features, shrunk toward the advertiser’s average click rate until it has impressions of its own, and the exploration slice gives it some auctions where it is close to the top but not at it. A few thousand impressions are enough to learn its rate, after which it competes on its own data. Thompson sampling over the uncertain estimate is the usual tool, and the logged probabilities keep that data usable for training and evaluation.