Design a Metrics and Monitoring System

Ingest 10 million samples a second from 100,000 hosts, keep them queryable for a year at shrinking resolution, draw dashboards in under a second, and page the right person within a minute.

Last updated 2026-09-30. Difficulty: hard. Patterns: time-series, compression, cardinality, downsampling, alerting. Reported at Datadog, Google, Amazon, Microsoft, Uber.

Functional requirements

• Collect metrics. Counters, gauges and histograms from hosts, containers and applications, each identified by a name and labels (service="api", region="eu-west-1").
• Query and dashboards. Aggregate across labels and over time ("p99 latency by region, last 6 hours"), rendered as graphs that refresh every 30 seconds.
• Alerting. Rules over queries ("error rate above 2 % for 5 minutes") that notify on-call through PagerDuty or Slack, grouped and deduplicated.
• Retention. Full resolution for two weeks, coarser rollups for a year.
• Multi-tenant. Many teams (or customers) share the platform with per-tenant limits.
• Out of scope. Logs and traces (mention how they link), anomaly detection models, and the dashboard UI.

Non-functional requirements

• Ingest scale (10 M samples/s · 100 M active series). 100 k hosts × ~1,000 series each, reported every 10 seconds.
• Alert latency (page within 1–2 min of the condition). This is the requirement that drives the hardest trade-off: alerting must keep working when the systems it watches, and parts of the monitoring system itself, are failing, so the alert path has to be simpler and more available than everything else.
• Query latency (p95 < 1 s for a dashboard panel). Recent data comes from memory; long ranges from pre-aggregated rollups, never from raw samples.
• Write availability (no gaps during an incident). Monitoring is most needed when everything else is on fire. Ingest must absorb bursts and degrade by dropping detail, not by going dark.
• Cost (~1.4 bytes per sample stored). At 10 M samples/s, a few bytes per sample is the difference between terabytes and petabytes a month.
• Isolation. One team adding a high-cardinality label must not take down ingestion or queries for everyone else.

Back-of-envelope estimates

• Samples per second: ~10 M. 100 k hosts × 1,000 series ÷ 10 s interval = 10 M samples/s.
• Raw size without compression: ~14 TB/day. A sample is a timestamp and a float, 16 bytes: 10 M × 16 B × 86 400 ≈ 13.8 TB/day. Storing that raw is not an option.
• Compressed size: ~1.2 TB/day. Delta-of-delta timestamps and XOR-encoded floats (Gorilla) average ~1.37 bytes per sample: 10 M × 1.37 × 86 400 ≈ 1.2 TB/day, about 12× smaller.
• Raw retention (15 days, 3 replicas): ~54 TB. 1.2 TB × 15 days ≈ 18 TB; with replication factor 3 in the ingest path and blocks deduplicated in object storage, roughly 54 TB of hot storage at peak, much less once blocks are compacted.
• Dashboard queries per second: ~7 k. 10 k dashboards open × 20 panels ÷ 30 s refresh ≈ 6.7 k queries/s, most for the last hour, which is served from memory.
• Alert rule evaluations per second: ~33 k. 1 M alert rules evaluated every 30 s = 33 k/s. Each is a query over the last few minutes; rules are sharded across evaluators.

Components

• Agents (on every host): Collect host and application metrics locally, pre-aggregate where possible, buffer to disk if the network is down, and send compressed batches every 10 seconds.
• Dashboards: Graphs that run queries over a time range and refresh every 30 seconds. The largest source of read load.
• Ingest gateway (auth · limits): Authenticates tenants, validates names and labels, enforces per-tenant rate and cardinality limits, and writes accepted samples to the ingest log, partitioned by series.
• Ingest log (Kafka · key = series hash): A durable buffer between agents and storage. Absorbs bursts, lets ingesters be restarted without losing data, and lets a second consumer (rules, anomaly detection) read the same stream.
• Ingesters (in-memory head + WAL): Each owns a shard of series. Appends samples to compressed in-memory chunks, writes a write-ahead log, serves queries for the last few hours, and every two hours cuts an immutable block.
• Series index (inverted index: label → series ids): Maps label pairs (service="api") to the series that have them, so a query can find its few hundred series among 100 M without scanning. Built per block and in memory for the head.
• Block storage (object storage · 2 h blocks): Immutable blocks of compressed chunks plus their index, in S3. Cheap, durable, and read by queries for anything older than a few hours.
• Compactor (merge · downsample): Merges small blocks into larger ones, removes the duplicate copies written by replicated ingesters, and produces 5-minute and 1-hour rollups (min, max, sum, count) for long-range queries.
• Query engine: Parses a query, picks a resolution from the time range, fans out to ingesters for recent data and to block stores for older data, merges, and aggregates. Splits long ranges into cached day-sized pieces.
• Query cache: Caches results per query and aligned time step, so a dashboard refreshing every 30 seconds only computes the newest interval.
• Rule evaluator (sharded by rule): Runs every alert rule on a fixed interval against recent data, keeps per-alert state (pending, firing, resolved), and sends firing alerts to the alert manager.
• Alert manager (dedupe · group · route): Deduplicates alerts from replicated evaluators, groups related ones into a single notification, applies silences and inhibitions, and routes to the owning team’s channel.
• PagerDuty / Slack: Where humans are notified. Outside our control and rate-limited, so the alert manager batches and retries.
• Meta-monitoring (independent stack): A small, separate monitoring setup that watches the monitoring system itself, including a "dead man’s switch" alert that must arrive every minute; if it stops, the alerting path is broken.

User flows

1. A sample travels from a host to storage. The write path that runs 10 million times a second: batch at the edge, validate and limit, buffer durably, compress in memory, flush immutable blocks.
   1. The agent sends a compressed batch of samples every 10 seconds. Batching and compression cut 10 M samples/s into ~10 k requests/s. If the network is down, the agent buffers to local disk for a while and replays later, so short outages leave no gaps.
   2. The gateway validates labels and enforces the tenant’s rate and series limits. A tenant over its active-series limit has new series rejected (existing ones keep flowing), and the response tells the agent which metric caused it. This is the main defence against cardinality explosions.
   3. Accepted samples are written to the ingest log, partitioned by a hash of the series’ labels. Hashing by series sends every sample of a series to the same partition and therefore the same ingester, which is what lets it compress consecutive samples together.
   4. An ingester appends each sample to its series’ in-memory chunk and to its write-ahead log. Timestamps are stored as delta-of-delta (usually zero for a regular 10 s interval, so one bit) and values as XOR against the previous value, averaging ~1.4 bytes per sample. A new series is added to the in-memory inverted index.
   5. Every two hours the ingester cuts an immutable block and uploads it to object storage. A block holds compressed chunks for its series and their index. Once uploaded, the ingester can drop that data from memory and its WAL segments.
   6. The compactor merges blocks, drops replica duplicates and writes rollups. Two-hour blocks are merged into daily and then two-week blocks, which makes long queries open far fewer files. 5-minute and 1-hour rollups store min, max, sum and count per series so any aggregation stays correct.
2. Load a dashboard. Find the right series through the index, read recent data from memory and older data from blocks at a resolution that fits the screen, and cache by time step.
   1. A panel asks for p99 latency by region over the last 6 hours.
   2. The cache returns everything except the newest step; only that piece is computed. Results are cached per query and aligned time step. A dashboard refreshing every 30 s recomputes 30 s of data, not 6 hours.
   3. The query engine uses the inverted index to find the matching series. Intersecting posting lists for service="api" and the metric name finds a few thousand series out of 100 M, without scanning anything.
   4. Recent hours are read from ingesters; older data from blocks, at a resolution chosen from the range. A 6-hour graph 1,000 pixels wide needs at most one point per ~20 s, so raw data is fine; a 90-day graph uses 1-hour rollups. Reading raw samples for a year-long graph would touch billions of points to draw a thousand.
   5. Partial results are merged and aggregated, and replica duplicates removed. Each series is held by three ingesters, so the engine deduplicates by series and timestamp. Aggregation (sum by region, quantiles) happens after merging.
3. Someone adds user_id as a label. The scale-breaking case in every metrics system: series count explodes, and with it memory, index size and query cost. Limits contain it to one tenant.
   1. A deploy adds a user_id label to a request counter. Each distinct label combination is a separate series. With 5 M active users, one metric becomes 5 M series per host group: more than the rest of the tenant combined.
   2. The gateway sees the tenant’s active series approaching its limit and rejects new series for that metric. Existing series keep flowing, so dashboards do not go blank. The team gets an error naming the metric and label, and an alert.
   3. Without limits, ingesters would run out of memory and the index would balloon. Every series costs memory for its head chunk and index entries (a few KB). Millions of new series in minutes can crash ingesters, which then lose their unflushed data for every tenant on them.
   4. Queries touching the metric would scan millions of series. Per-query limits on series and samples fail such queries quickly instead of tying up the query fleet.
4. An ingester crashes. The failure path: replicated writes keep queries complete, and the WAL plus the ingest log bring the crashed shard back without gaps.
   1. One ingester dies with two hours of unflushed data in memory. Its series are also held by two other ingesters (replication factor 3), so queries still see every sample; the engine just gets results from two replicas instead of three.
   2. A replacement starts, replays its write-ahead log, then resumes consuming from its last committed offset. The WAL restores the in-memory chunks; Kafka fills the gap between the WAL and now. Offsets are committed only after samples are in the WAL, so nothing falls between them.
   3. Meta-monitoring notices the ingester was down and records the event. The monitoring system cannot reliably watch itself, so a small independent stack scrapes its health.
   4. The compactor later removes the duplicate copies in uploaded blocks. Three ingesters upload three copies of the same series; vertical compaction merges them into one, so storage does not triple in object storage.
5. An alert fires and someone is paged. The background path that matters most: evaluate, wait out blips, deduplicate, group, route, and prove the path itself is alive.
   1. Every 30 seconds the rule evaluator runs "5xx rate above 2 % for 5 minutes" for each service. Rules are sharded across evaluators by hash, and each shard runs on two evaluators for redundancy. They query only recent data, which is in ingester memory.
   2. The condition holds; the alert goes pending, then firing after five minutes. The "for" duration filters out blips. Each alert’s state is kept per label set (service="checkout"), so different services fire independently.
   3. Firing alerts are sent to the alert manager, which deduplicates, groups and routes them. Both evaluators send the same alert; the alert manager keeps one. Twenty services failing because of one database become one grouped notification. Silences and inhibitions (do not page for symptoms when the cause already paged) apply here.
   4. The owning team is paged. Routing uses labels (team="payments"). Notifications are retried and repeated until acknowledged.
   5. A heartbeat alert that always fires proves the path works; its absence pages someone. The dead man’s switch: an alert that fires every minute and is routed to an external service that pages if it stops arriving. It catches the failure mode where alerting is silently broken.

Deep dives

The theory behind it

• Observability, operations and rollouts: Metrics, logs and traces; SLIs, SLOs and error budgets; alerting; safe deploys (canary, blue-green, feature flags); migrations without downtime; capacity planning; and how to answer "how would you know it is working".
• Batch and stream processing: Stream versus batch, windowing and watermarks, late and out-of-order data, exactly-once counting, approximate algorithms, and the lambda/kappa argument in one paragraph.
• Object storage and large files: Blob storage, pre-signed uploads, chunking and resumable transfer, multipart, erasure coding, storage tiers and lifecycle, plus how media pipelines are structured.

Related

• Run this question for a friend: the interviewer kit