Design a Price Tracker

Track 500 M products' price history and alert 50 M users when a price drops below their target, in time to buy.

Last updated 2026-10-26. Difficulty: hard. Patterns: adaptive-scheduling, change-detection, time-series, range-queries, alerting. Reported at Amazon, Google, Meta, Microsoft, Shopify, Instacart.

Functional requirements

• Watch a product with a target price. From a pasted link or the extension's button. The link is canonicalised to a listing (marketplace plus product id) and the target is in that listing's currency.
• Price history chart. Every change since we started tracking, per series (new, used), with gaps where the price was unknown.
• Drop alerts by push or email. When the price reaches the target, fast enough to buy before a deal ends. Once per crossing, not on every check while the price stays low.
• Browser extension. Shows the chart on the product page and a one-click watch. Opt-in: reports the price the page shows, which we treat as a hint, never as the truth.
• Manage the watch list. List, edit the target, pause, delete. An edit takes effect at the next price change, not at the next batch run.
• Multiple marketplaces. The same product on amazon.com and amazon.de is two listings, with separate prices, currencies and histories. Conversions are for display only.
• Out of scope. Crawling mechanics (the existing crawler), notification delivery (the existing notification system), price prediction, deal forums and affiliate accounting.

Non-functional requirements

• Fixed check budget (~15 k checks/s). Set by the retailers' API quotas and by crawl politeness, not by us. This is the requirement that drives the hardest tradeoff: every check spent on one listing is freshness taken from another.
• Alert latency once a change is seen (p95 < 60 s). From the observation to the hand-off to the notification system. The bigger term is detection lag, set by the check interval: 2 to 5 min for the hot tier, about an hour warm, a day for the long tail.
• Alert correctness (one per crossing · no glitch alerts). Exactly one alert per watch per crossing, through retries and crashes, and never on an implausible price until a second source has confirmed it. A false $1 alert to a million people costs more trust than a missed deal.
• Scale (500 M listings · 1 B watches). 50 M users with ~20 watches each. Heavily skewed: the median watched listing has 2 or 3 watches, the hottest ~2 M.
• Chart latency (p99 < 200 ms). The extension draws on the retailer's page while it loads; a slow chart feels broken. One partition read behind a CDN.
• Honest history (append-only, gaps explicit). The chart is what users trust. Never lose a change point, record the source of each, and show unknown periods (out of stock, not checked) as gaps rather than a straight line.
• Extension privacy (product pages only). Reports carry an install id, not a user id, and come only from supported product pages. No browsing history leaves the browser.
• Availability (99.9 %). When a source is down, checks queue and alerts are late. The pipeline never guesses a price to stay on time.

Back-of-envelope estimates

• Checks per second (the budget): ~14.4 k/s. Tiered by value: 1 M hot listings every 5 min = 3.3 k/s, 20 M every hour = 5.6 k/s, the other 479 M once a day = 5.5 k/s. Total ≈ 14.4 k/s, inside a ~15 k/s budget set by API quota and crawl politeness, with the rest kept for verification checks.
• Uniform interval at that budget: ~10 h. 500 M ÷ 14.4 k/s ≈ 34.7 k s ≈ 9.6 h between checks of each listing if the budget were spread evenly. A 6-hour lightning deal could start and end between two checks. Scheduling by value is not an optimisation here, it is the product.
• Product API calls: ~1 k calls/s. About 70 % of checks go to marketplaces with a product API: 10 k items/s. At 10 items per call that is 1 k calls/s; the other ~4.4 k/s are page fetches through the crawler. Batching is what makes the quota go ten times further.
• Price changes: ~600/s · ~5 k/s peak. 14.4 k × 86 400 ≈ 1.24 B checks a day; about 4 % find a change, so 50 M changes/day ≈ 580/s. On Prime Day a few hundred thousand a minute: 300 k ÷ 60 = 5 k/s, almost 9× normal and a third of all checks. Each change is one range read in the matcher, so the matcher is sized for the peak.
• Price history growth: ~240 GB/yr. Changes only: 50 M/day × 365 × 13 B (timestamp, price in minor units, flags) ≈ 240 GB a year, ~36 points per listing per year, so a 5-year chart is ~2.4 KB. Storing every observation would be 1.24 B × 13 B × 365 ≈ 5.9 TB a year, 25× more, and almost all of it repeats.
• Watch index: ~60 GB. 1 B watches × ~60 B (listing, target, watch id, user id, state, epoch) ≈ 60 GB before replication, 180 GB with three copies. Small in total; the problem is skew, not size: the median watched listing has 2 or 3 watches, the hottest ~2 M.
• Hot product fan-out: 600 k alerts in ~30 s. The hottest listing has ~2 M watches; a drop from $249.99 to $169.99 crosses ~30 % of the targets, 600 k. The range read is ~36 MB across 32 sub-partitions, a second or two. At a 20 k/s producer quota on the notification system the sends take 30 s: pacing, not reading, sets the time.
• Chart reads: ~1.2 k/s. Say 5 M active extension users each open ~20 product pages a day: 100 M reads/day ≈ 1.2 k/s, peaking 5–10× on sale days. Each is one ~2.4 KB partition read. The same page views can send up to 1.2 k price reports a second, free signal worth ~8 % of the check budget.

Components

• Retailers (product APIs · product pages): Amazon-style marketplaces and other large stores. Their product APIs are structured but quota-limited and can lag the site by a minute or two; their pages show what a shopper sees but change layout without notice. We are a guest on both, so they set the budget, not us.
• Price fetchers (API batches · crawler jobs): Stateless workers that turn due checks into observations. Where a marketplace has a product API they batch 10 listings per call within the quota; otherwise they hand a page fetch to the existing crawler, which owns politeness, proxies and rendering, and run a versioned per-site extractor on the HTML. Every observation carries its source and extractor version.
• Check scheduler (next-check time · budget): Holds a next_check_at for each of 500 M listings in sorted sets sharded by listing id, and drains due listings into per-source token buckets: the product API per marketplace, the crawler per site. After each check it recomputes the interval from the listing's value (armed watches, gap to the nearest target, change rate, sale boost) and one global factor that keeps total demand at the ~15 k checks/s budget. Verification requests jump the queue.
• Notification system (existing · push + email): The existing notification platform: channel routing, preferences, provider retries and per-hop idempotency are its job, not ours. We send one request per alert with the idempotency key hash(watch_id, arm_epoch), high priority, and a TTL so a delayed push for an ended deal is dropped. Our producer quota on its high-priority lane, ~20 k/s, is what paces a hot listing’s fan-out.
• Browser extension (chart on the product page): Recognises a supported retailer's product page, maps it to a listing id and draws our price history on the page, with a one-click watch button. With the user's opt-in it reports the price the page shows, signed with a per-install key and with the page context (signed in, member price, coupon shown). It sends nothing from pages that are not product pages on supported sites.
• Latest price (KV · price, seq, last checked): One small record per listing (~40 bytes, ~20 GB in all): current price, stock and offer flags, a sequence number, last_checked and any pending suspect price. The detector updates it with compare-and-set on the sequence, so two racing observations of one listing never both publish a change. Replicated and persisted: losing it would make every next observation look like a change.
• Change detector (dedupe · plausibility): Every observation, from a check or an extension report, comes here and is compared with the latest price. Identical observations (~96 %) only update last_checked; a real change is appended to history and published; a suspicious drop is held as pending and sent back for confirmation from another source. A crowd report never becomes a change on its own, only a verification request.
• Listing events (Kafka · key = listing id): One topic carrying both price.changed from the detector and watch.created, updated and deleted from CDC, partitioned by listing id. The shared key is the point: for any listing, price changes and watch changes reach the matcher in one total order, so a new watch cannot slip past a drop. About 600 events/s normally, ~5 k/s at the Prime Day peak.
• Alert matcher (range match · watch state): Consumes listing events and is the only writer of the watch index. On a drop from p_old to p_new it reads the watches with target in [p_new, p_old) and fires the ARMED ones; on a rise it re-arms MET watches that the price has cleared by the 2 % band. Large crossings go to a wave sender that paces them under the notification quota, and after every event it reports the listing’s armed count and nearest target to the scheduler.
• Web + app (watch list · alerts): Where users paste a product link, set a target and manage their watch list, and where an alert lands when it is tapped. Shows the current price, the target and the chart. Targets are entered in the listing's own currency; any converted figure is labelled as an estimate.
• API gateway (auth · limits · CDN for charts): Authenticates users and extension installs, rate limits reports per install and per IP, and routes watch calls to the watch service, chart reads to the history store and reports to the change detector. Chart responses are public with a 60-second max-age and an ETag from the history sequence, so a CDN absorbs the herd on a hot listing.
• Price history (change points · per listing): Only changes, not observations: partition (listing_id, series, year), one row per change point with time, price in minor units, flags and source. About 36 rows per listing per year, ~240 GB a year in total, and a 5-year chart is one ~2.4 KB range read. Unknown periods (out of stock, not checked) are explicit points, so a chart never draws a line through a gap.
• Watch index (by listing, sorted by target): A second copy of every watch, partitioned by (listing_id, sub) and clustered by (target_minor, watch_id), with state (ARMED or MET) and arm_epoch. A price change becomes a range read whose size is the number of watches it crosses. Listings with more than 50 k watches are split into 32 sub-partitions by hash(watch_id) and read in parallel.
• Watch service (canonicalise · validate): Turns a pasted URL into a canonical listing id (marketplace plus product id, tracking parameters dropped), validates the target against the listing's currency, and creates, edits and deletes watches. It writes only the user-keyed watch store and never touches the index the matcher reads, which is what keeps watch changes and price changes in one order.
• Watch store (by user · source of truth): Every watch keyed by (user_id, watch_id): listing, series, target in minor units, channels. About 1 B rows and ~150 GB, sharded by user so a watch list is one partition. Change data capture publishes every create, edit and delete onto the listing's event stream, keyed by listing id.

User flows

1. A scheduled check finds a price drop. The core path from a due check to an alert. It shows why only changes are stored, why the alert is a range read over targets rather than a scan of watches, and how the result sets the next check.
   1. The scheduler takes the listings due this minute and spends the marketplace's API tokens on them. Due listings come out of the sorted sets oldest first, and a listing only leaves when its source has a token, so a quota cut stretches intervals instead of producing 429s. A check not reported back within 2 minutes is dispatched again; a duplicate costs one item of quota and the detector drops it as identical.
   2. A fetcher batches ten due listings from one marketplace into one product API call. Batching is what stretches the quota: 1 k calls a second is 10 k items. Listings are grouped by marketplace because quotas and currencies are per marketplace. A 429 puts the batch back with a short delay and lowers that marketplace’s token rate; nothing is dropped.
   3. The fetcher turns each item into an observation and sends it to the change detector. An observation is the fact plus its provenance: listing, series, price in minor units, currency, stock, source, extractor version and observed_at. Observations are routed by hash of listing id, so one detector instance usually sees a listing in order and keeps its latest price warm in memory.
   4. The detector compares with the latest price, sees $249.99 become $169.99, and advances the record with compare-and-set. This is the dedupe: about 96 % of observations match the latest price and stop here after touching last_checked. A 32 % drop is inside this listing’s range (its 90-day low is $179.99 and a sale is on), so it is not suspect. The sequence number makes racing observations safe: only one of them advances seq 42 to 43, and an observation older than the latest is ignored.
   5. The detector appends one change point to the listing's history. One row with ~13 bytes of payload, in the partition for this listing, series and year. The row key includes the timestamp, so a retried append rewrites the same cell instead of adding a second point. Nothing at all is written for the thousands of checks that saw the same price.
   6. The detector publishes price.changed on the listing's event stream. Keyed by listing id, the same key the watch CDC uses, so the matcher sees this change and any watch edit for the listing in one order. The event carries the old price as well as the new one: the matcher needs the interval between them, not just the new value.
   7. The matcher reads the watches whose target lies between the new and the old price. Targets at or above $249.99 were already met at the old price, and targets below $169.99 are still not met. Only [169.99, 249.99) can have crossed, so the read touches exactly those rows, sorted by target. Rows in state MET (alerted earlier and not yet re-armed) are skipped.
   8. For each armed watch the matcher sends an alert with a deterministic key, then marks the watch MET. The key is hash(watch_id, arm_epoch), so replaying this event after a crash produces the same keys and the notification system drops the repeats. That is why sending before recording MET is safe; recording first would risk watches marked MET whose alert never left. Channel choice, quiet hours and provider retries belong to the notification system.
   9. The matcher reports the listing's armed count and nearest target, and the scheduler sets the next check. With 1,812 watches now MET, 310 armed watches remain below the price, the nearest at $159.99, 6 % away; and a change in the last hour boosts the value, because repricing comes in bursts. The scheduler recomputes clamp(k ÷ value, 2 min, 24 h) and lands on about 4 minutes.
2. The extension shows history on a product page. The main read path, and the one place users feed us prices. The chart is a single partition read behind a CDN; the report is a hint that can cost at most one verifying check.
   1. A shopper opens a product page on amazon.de and the extension recognises the listing. The listing id comes from the page: the marketplace from the domain, the product id from the URL or the page’s structured data. The extension is inert everywhere else, and never sends a URL or anything from a page that is not a supported product page.
   2. The extension requests the listing's history for the last year. The response is small and the same for everyone, so it is cacheable at the edge for 60 seconds, with the listing’s history sequence as the ETag. The chart has to render well inside the page’s own load time or the extension feels broken.
   3. On a cache miss the API reads the year's change points from one partition. About 36 rows for a typical listing and a few KB even for a volatile one. Long ranges on listings repriced many times a day read a daily min/max rollup instead of every point.
   4. The page shows €164.99 but our latest price is €179.99, so the extension prepares a report. The extension reads the price with the same per-site rules our extractor uses, plus the context that makes a price personal: signed in, member-only badge, coupon checkbox, delivery country. A price that differs only because of a coupon or a member discount is not reported.
   5. The extension posts a signed report, which the API forwards to the change detector. The report is tied to an install, not a user, and rate limited per install and per IP at the gateway. Its weight comes from the install’s track record: how often its past reports were confirmed by a check.
   6. The detector does not trust the report; it asks the scheduler for a verification check now. A report is never written to history or published as a change, so a spoofed one can only cost a check. Report-triggered checks are capped at one per listing per 10 minutes, and the latest record notes that one is pending, so a hundred matching reports cost the same single check.
   7. The verification check confirms €164.99 through the product API, and it becomes an ordinary change. From here it is the drop flow: compare, append, publish, match. The crowd made the alert faster by however long the listing would have waited for its next scheduled check, which for a long-tail listing could be most of a day.
3. Prime Day: one drop, 600 k alerts. The scale-breaking case: one listing with ~2 M watches drops below 600 k targets at once. The fan-out must finish in about a minute, stay under a shared notification quota, and stop if the price rebounds.
   1. The hot listing is checked every 2 minutes; this check returns $169.99, down from $249.99. Its value is at the cap, with 2 M armed watches and a sale under way, so it sits at the 2-minute floor. Checking faster buys little, because the API’s own data can be a minute or two behind the site.
   2. The detector accepts the drop, appends it and publishes price.changed. 32 % is a big step, but the listing is flagged as on sale and $169.99 is close to its 90-day low, so it is not suspect. Had it read $16.99, the matcher would not see it until a second source confirmed it.
   3. The matcher reads the crossed range from all 32 sub-partitions in parallel. 600 k rows of ~60 bytes is ~36 MB, about 19 k crossed rows per sub-partition, back in a second or two. Sub-partitions are by hash of watch id rather than by price band, because targets cluster at round numbers like $199.99 and one band would hold most of them.
   4. The matcher hands the crossing to a wave sender, which collapses it per user and shuffles it. Handing off keeps the listing’s Kafka partition moving, so the other listings on it are not delayed by this fan-out. Collapsing sends one message to a user who watches three colours of the same product. The shuffle is seeded by hash(watch_id, seq), so the order differs on every drop and nobody is always last.
   5. The sender sends waves at the 20 k/s notification quota and asks the scheduler for a re-check every 30 seconds. 600 k at 20 k a second is 30 s. Each message has a 30-minute TTL, so a push stuck behind a provider outage is dropped instead of arriving after the deal. Watches are marked MET as each wave is accepted.
   6. A re-check finds $199.99; the rise arrives on the same stream and the remaining waves skip targets below it. The sender reads the listing’s current price before each wave. Watches with targets under $199.99 are no longer met, so they are not sent and stay ARMED for the next real crossing; those between $199.99 and $249.99 still get theirs. The rise also runs the re-arm range for any MET watch it lifted clear of the band.
   7. Users who tap the alert and open the chart are served from the CDN. The push deep-links to the retailer, but a share of users open our chart first: thousands of requests a second for one listing. A 60-second edge cache turns that into about one history read a minute per edge location.
4. A $1 price that is not real. A failure path: an extractor misreads a page and reports a $299 TV at $1.00. The design must not alert on it, must not draw it on the chart, and must notice that the extractor is broken.
   1. A scheduled check for a store without an API goes through the crawler, and the extractor reads $1.00. The site moved its add-on offers above the main price, and the extractor’s selector now matches a $1.00 cable. Nothing about the fetch failed: status 200, valid HTML, a well-formed price. This is the failure scraping produces.
   2. The fetcher sends the observation with its source and extractor version. Provenance on every observation is what makes the next steps possible: the detector can pick a different source to confirm with, and a bad extractor version can be found and backed out later.
   3. The detector flags the price as suspect and holds it as pending, without touching history or publishing. Suspect means more than 50 % below the 90-day low, more than 70 % in one step, or a currency, unit or offer mismatch; here it is 99.7 % below. The latest record keeps $299.00 as the current price and $1.00 as pending, so the chart, the matcher and the API all still see $299.00.
   4. The detector asks the scheduler for a confirmation from a different source within a minute. A different source matters: running the same broken extractor again would confirm the same wrong number. For a site without an API the alternative is the previous extractor version or the mobile page, which has a different layout.
   5. The confirmation reads $299.00, so the pending price is discarded and counted against extractor v14. Nobody was alerted and the chart never showed $1.00. Had the confirmation also read $1.00, it would be a real pricing error: written and published as confirmed, about a minute later than an unsuspicious drop.
   6. Contradictions reach 2 % of v14's observations within an hour, so the detector quarantines it and the scheduler routes the site to the fallback. One wrong price is noise; a rate is a broken extractor. Quarantine means its observations are ignored and the site’s checks use the fallback extractor until v14 is fixed. The same signal pages the team that owns the site’s rules, usually within minutes of a retailer’s redesign.
5. Add a watch, fire, and re-arm. The asynchronous path that keeps the index right. A new watch travels through CDC on the same ordered stream as prices, so it cannot miss a drop that races it, and a crossing alerts once and re-arms only after the price clearly recovers.
   1. A user pastes an amazon.de link and sets a target of €150. The watch service canonicalises the URL to amazon.de:B0C1234567, dropping tracking and affiliate parameters, and stores the target in euros because that listing charges euros. The add screen already shows the chart and current price, so the UI can say at once if the target is already met.
   2. The watch service writes the watch to the user-keyed store. This store is the source of truth for what the user asked for and the only thing the watch service writes. It does not touch the watch index, which is what keeps the next step’s ordering guarantee simple.
   3. CDC publishes watch.created onto the listing's event stream, keyed by listing id. Same topic and key as price.changed, so the matcher sees this watch and any price change for the listing in one order. CDC rather than a second write from the service means the event exists if and only if the watch committed.
   4. The matcher inserts the watch into the index and evaluates it against the price it last saw for the listing. €150 is below the current €179.99, so it goes in ARMED with epoch 1. Had a drop to €149 arrived just before this event, the matcher would already hold €149, insert the watch and fire it at once: the race between creating a watch and a price change cannot lose the alert.
   5. The matcher tells the scheduler that the listing now has a closer target. The nearest armed target is now 17 % below the price instead of 30 %, so the value rises and the next check comes sooner. A listing with no known price at all is checked at once, because the user is looking at it.
   6. Weeks later the price falls to €149 and the watch fires; then it rises to €165 and the matcher re-arms it. The drop fires with key hash(watch_id, 1) and sets MET. While the price wobbles between €149 and €152 nothing happens, because re-arming needs a price above target × 1.02 = €153. The rise to €165 reads the re-arm range [149 ÷ 1.02, 165 ÷ 1.02) = [€146.08, €161.76), finds this watch MET and sets it ARMED with epoch 2, so the next drop is a new alert with a new key.

Deep dives

The theory behind it

• Time-series data: Storing and querying time-series data such as metrics, sensor readings, prices and events: write patterns, time-partitioning, compression, downsampling and retention, and choosing between time-series databases, wide-column stores and OLAP engines.
• Web crawling at scale: How large crawlers fetch billions of pages: the URL frontier, politeness per host, prioritisation and recrawl scheduling, DNS and fetching, URL normalisation and deduplication, content fingerprints, robots.txt, traps and distributing the crawl.
• Hot keys, hot partitions and data skew: Why real traffic is never uniform, how a single hot key or partition takes down a sharded system, and the fixes: key splitting, local caching, request coalescing, write buffering, salting, adaptive rebalancing, and skew in joins and stream processing.

Related

• Design a Metrics and Monitoring System
• All 83 system design interview questions
• Run this question for a friend: the interviewer kit