Design a Distributed Logging System

• Collect logs from every host: Application logs (stdout and files, mostly JSON, plenty of free text and stack traces) and infrastructure logs (syslog, Kubernetes, load balancer access logs) from 100 k hosts, each line labelled with service, environment, cluster and host.
• Search by text and fields: For example {service="checkout", env="prod"} |= "card declined" | json | status >= 500. Results come newest first, and a line is searchable within seconds of being written. Most searches name a service and a time range; some hunt for one trace id across everything.
• Live tail: Stream new matching lines for a service as they are written, like tail -f across 2,000 pods, within about two seconds.
• Saved searches that alert: “More than 50 payment timeouts a minute in checkout for 5 minutes” notifies the owning team. The same counts feed dashboards.
• Retention in tiers: 30 days searchable interactively, a year retained for audits and security investigations, with per-tenant policies and legal holds.
• Many tenants, with quotas and cost: Hundreds of teams share the platform. Each has ingest quotas, sees its own volume and cost, and can only read its own logs.
• Parsing and redaction: JSON passes through; known formats (nginx, syslog) are parsed; secrets and personal data are masked before anything is stored.
• Out of scope: Metrics and trace storage (logs link to traces by trace id), the search UI, SIEM correlation rules, and machine-learned anomaly detection beyond a mention.

• Ingest scale (10 M lines/s peak · ~1 PB/day raw): 100 k hosts at ~100 lines a second each, ~1 KB per line. Volume jumps 2 to 3× during incidents, exactly when logs are most needed.
• Never hurt the application (logging never blocks a request): A slow or dead pipeline must cost the application nothing but a growing local buffer. The agent runs under a hard CPU and memory cap.
• Freshness (searchable p99 < 10 s after write): During an incident engineers search for what happened seconds ago. Live tail needs about 2 s.
• Query latency (p95 < 3 s · one service, last hour): Searches over 30 days may take tens of seconds and stream results as they come; searches over data older than 30 days run as background jobs.
• Durability (no acknowledged line lost): At-least-once from the host to storage, with batch ids so retries do not duplicate. Lines are lost only by an explicit, counted policy (dropping debug or sampling info under quota), never silently.
• Cost (storage < 1 cent per raw GB for a year): This is the requirement that drives the hardest trade-off. An inverted index on every field makes any search fast but needs ~60 PB of SSD for 30 days; indexing only labels and scanning compressed chunks is ~25× cheaper for a whole year but makes broad searches slow.
• Tenant isolation (one team’s flood cannot delay another): Per-tenant quotas, shuffle-sharded partitions and fair query queues, so a debug flag left on in one service does not degrade logging for everyone.
• Privacy (no secrets or raw PII at rest): Card numbers, tokens and emails are masked on the host and checked again at ingest; access is per tenant and every search is audited.

• Ingest rate: 10 M lines/s · 10 GB/s raw. 100 k hosts × ~100 lines/s each at peak = 10 M lines/s. At ~1 KB per line that is 10 GB/s (80 Gbit/s) of raw text; zstd at the agent cuts what crosses the network to about 1 GB/s.
• Raw volume per day: ~0.9 PB (plan for 1 PB). 10 GB/s × 86 400 s ≈ 864 TB/day. Assuming the peak holds all day leaves headroom for growth; the rest of the sizing uses 1 PB/day.
• Compressed volume per day: ~100 TB. Log lines repeat the same templates, so zstd over a batch from one stream compresses about 10×: 1 PB ÷ 10 = 100 TB/day, or about 25 M chunks of ~4 MB.
• Index every field for 30 days: ~60 PB of SSD. An inverted index on every field plus the stored line takes roughly the raw size per copy (anywhere from about half the raw size to more than it, depending on mappings and compression): 1 PB × 30 days × 2 copies ≈ 60 PB of SSD, plus hundreds of indexing nodes. This is the number that rules out indexing everything.
• Chunk storage by tier: hot 0.3 PB · warm 3 PB · cold ~34 PB. At 100 TB/day compressed: 3 days on NVMe = 300 TB; 30 days in object storage = 3 PB; days 31 to 365 in an archive class = 335 × 100 TB ≈ 34 PB. At roughly $100, $23 and $4 per TB-month that is about $30 k + $69 k + $134 k ≈ $233 k a month for a full year, against ~$6 M a month for 30 days of SSD index.
• Buffer for a day of catch-up: ~260 TB · 512 partitions. About 1 GB/s compressed into Kafka × 86 400 s ≈ 86 TB, × 3 replicas ≈ 260 TB. At ~10 MB/s per partition the floor is 100 partitions; 512 leaves room for more than a hundred ingesters and per-tenant shuffle sharding.
• Scan for one service-hour: ~360 GB raw · ~1.5 s on 500 cores. A big service such as checkout is ~1 % of volume: 10 GB/s × 1 % × 3 600 s = 360 GB raw (36 GB compressed). Decompressing and substring matching runs at ~0.5 GB/s per core, so 720 core-seconds, ~1.5 s across 500 cores. The same hour across every service is 36 TB: scanning is affordable only when a search names its labels or a bloom filter skips the chunks.
• Agent disk buffer per host: ~28 hours. An average host writes 100 lines/s × 1 KB = 100 KB/s, ~10 KB/s compressed. A 1 GB buffer holds 1 GB ÷ 10 KB/s = 100 000 s ≈ 28 hours, so a pipeline outage of a few hours loses nothing; a noisy host at 10× the average still gets nearly 3 hours.

• Host agents (tail · disk buffer · scrub): One daemon per host (Fluent Bit, Vector or an OpenTelemetry Collector) tails application and system log files, stitches multi-line stack traces, attaches labels, masks secrets and PII, and ships zstd-compressed batches with a batch id. Applications only ever append to local files, so a slow pipeline never blocks them. When the gateway pushes back, batches wait in a capped disk buffer and the agent sheds debug first, then samples info, and counts what it dropped.
• Engineers (UI · CLI): Search, live tail and saved searches from a web UI or CLI, signed in with SSO and authorized per tenant. During an incident hundreds of them run near-identical searches at once, which is the read-side peak to plan for.
• Ingest gateway (auth · quotas · 429): Authenticates each agent to a tenant, checks labels against the tenant’s stream limit, charges its byte quota, and writes the batch to the buffer with acks=all before answering 202. Over quota it returns 429 with a retry-after instead of queueing, so backpressure lands on the agent’s disk, not in gateway memory. Emits bytes accepted and rejected per tenant, service and level as usage metrics.
• Log buffer (Kafka · 24 h · 512 partitions): A partitioned log between collection and indexing, keyed so a stream stays on one partition and each tenant is shuffle-sharded onto a few partitions. Holds 24 hours (~260 TB with three replicas), so indexing can fall behind during an incident and catch up while ingest never slows. Three consumer groups read it independently: ingesters, live tail and the saved-search evaluator.
• Saved-search evaluator (streaming match → counts): A consumer group that matches every line against ~50 k saved searches and counts matches per search per minute. Each line is examined once as it passes, instead of thousands of searches re-scanning the last five minutes every minute. Searches that need parsing and grouping over long windows run as scheduled queries instead.
• Metrics and paging (existing monitoring stack): The company’s metrics system receives log-derived counts and usage meters. Its alert rules, grouping, routing and dead man’s switch page the owning team, so the logging system does not build a second pager.
• Ingesters (head chunks · blooms): Each owns a few partitions. Appends lines to an in-memory head chunk per stream (searchable at once), drops retried batches by their sequence number, and flushes a chunk at ~4 MB compressed or one hour, with a token bloom filter. Kafka is its write-ahead log: the committed offset is the oldest one still in an unflushed chunk, and each flushed chunk records its last offset, so a replay never writes a line twice.
• Chunk storage (object storage · 30 days): The source of truth for every flushed chunk and its bloom filter, at keys like {tenant}/{day}/{hour}/{stream}/{chunk}. About 3 PB for 30 days at roughly $23 per TB-month. Time-partitioned prefixes make retention a prefix delete and per-tenant policies trivial.
• Live tail (own consumer group · WebSocket): Tail workers read every partition once from the head of the buffer and match each line against the active tail sessions, indexed by tenant and labels; matches go to the node holding the engineer’s WebSocket. Independent of indexing, so tail stays live while ingesters are behind. Each session is capped at 500 lines a second and reports how many it skipped.
• Chunk index (labels → chunks, per tenant-day): For each tenant and day: label pairs to streams, and streams to chunk references with time range, size and last offset. About 200 bytes per chunk, ~5 GB a day for the whole company against ~1 PB of raw logs. Partitioned by tenant and day, so retention drops a whole index partition, never individual rows.
• Lifecycle job (tier · expire): Daily, per tenant: moves day partitions to the archive class at day 30, deletes chunk prefixes and index partitions when the tenant’s retention ends, and skips anything under legal hold. Works on whole tenant-days, so a year of retention is a few thousand prefix operations, not billions of row deletes.
• Query frontend (split · budgets · fair queues): Parses the query, splits its time range into hourly sub-queries, estimates the bytes to scan from the index and refuses or defers anything over the tenant’s budget. Dispatches the newest hours first with bounded parallelism, merges results in time order, and stops as soon as it has the requested number of lines. Queues per tenant, so one team’s incident cannot starve another’s.
• Queriers (stateless · scan): Stateless workers that run sub-queries: find chunks in the index, check bloom filters for rare tokens, fetch surviving chunks from the hot cache or object storage, decompress, apply cheap substring filters before any parsing, and stream matches back. About 0.5 GB/s of raw text per core; the fleet autoscales on queue wait, because query load rises 5 to 10× during incidents.
• Hot chunk cache (NVMe · last 3 days): A read-through NVMe cache on the querier nodes, about 300 TB, pre-warmed with newly flushed chunks and blooms. Most searches look at the last hours or days, so they read local SSD in about a millisecond instead of tens of milliseconds from object storage. Losing it costs latency, never data.
• Cold archive (archive class · 1 year): Chunks older than 30 days in an archive storage class at a few dollars per TB-month, about 34 PB. The index and bloom filters for this data stay in standard storage, so a cold search can work out which chunks it needs and restore only those, in minutes to hours.

1. A log line goes from a host to searchable: The write path that runs 10 million times a second: append locally, batch and buffer on the host, acknowledge only once the line is durable in the buffer, build compressed chunks per stream, and store them in time partitions that can be tiered and dropped whole.
   1. The application writes a JSON line to stdout, and the container runtime appends it to a file on the host.
   2. The agent tails the file, stitches multi-line stack traces, adds labels, and masks secrets and personal data.
   3. The agent batches lines per stream for up to 1 s or 1 MB, compresses them, and sends them with a batch id.
   4. The gateway authenticates the tenant, charges its quota, and writes the batch to the buffer before acknowledging.
   5. An ingester consumes the partition, drops batches it has already stored, and appends lines to the stream’s in-memory head chunk.
   6. When a chunk reaches about 4 MB compressed or an hour old, the ingester builds a bloom filter of its tokens and uploads both to object storage.
   7. The ingester records the chunk in the tenant’s daily index, then commits its offset in the buffer.
   8. As the data ages, the lifecycle job moves the tenant’s day partitions to the archive tier and deletes them when retention ends.
2. Find a trace id in a day of logs: The read path: narrow by labels and time with a tiny index, skip almost every chunk with bloom filters, scan the survivors in parallel from SSD or object storage, include what is still in ingester memory, and stop as soon as the answer is complete.
   1. An engineer searches checkout’s production logs over the last 24 hours for a trace id, newest first, limit 100.
   2. The frontend splits the range into hourly sub-queries, newest first, and checks the cost against the tenant’s budget.
   3. Queriers look up which of checkout’s chunks overlap each hour.
   4. Queriers check each chunk’s bloom filter for the trace id; about 99 % of chunks are skipped.
   5. Queriers fetch the surviving chunks from the SSD cache, or object storage on a miss, then decompress and grep them in parallel.
   6. For the newest hour, queriers also ask the ingesters for lines still in head chunks.
   7. The frontend merges results newest first and stops once it has 100 lines, cancelling the sub-queries still running.
3. One service floods its logs: The scale-breaking case: a deploy turns on debug logging and one service’s volume rises 20×. Quotas, shedding at the source and shuffle sharding keep the damage inside that tenant, and every dropped line is deliberate and counted.
   1. A bad deploy turns on debug logging in checkout, and its volume jumps 20× to 2 GB/s raw.
   2. The gateway charges the payments tenant’s byte quota and answers 429 once it is used up.
   3. Agents keep rejected batches on disk and, as the buffer fills, drop debug first, then sample info, and never drop warn or error.
   4. The gateway records the tenant’s accepted and rejected bytes as usage metrics, and the owning team is alerted.
   5. Other tenants’ batches flow normally, because their partitions and ingesters are mostly different ones.
   6. Searches during the incident are queued per tenant and capped by bytes scanned.
   7. After the fix, agents drain their disk buffers at the quota rate, oldest first.
4. Indexing falls behind and catches up: The failure path: an ingester crashes with unflushed chunks, and later the whole indexing tier falls half an hour behind. The buffer absorbs both; ingest never slows, nothing is lost or duplicated, and tail and alerts stay live.
   1. An ingester dies with 40 minutes of unflushed head chunks in memory.
   2. Its partitions are reassigned, and the new owner resumes from the last committed offset.
   3. During replay, batches already in flushed chunks are skipped using each stream’s flushed offset.
   4. Live tail and saved-search alerts keep running, because they read the buffer with their own consumer groups.
   5. Later, a region-wide incident triples log volume and every ingester falls 30 minutes behind; the buffer absorbs it.
   6. Ingesters catch up by consuming faster than real time, and more ingesters are added up to the partition count.
   7. Until then, search results for the lagging window carry a warning naming the partitions and their lag.
5. Live tail and a saved-search alert: The streaming paths. Both read the buffer from its head with their own consumer groups and do their work once per line as it passes, instead of polling the index, so they are live within seconds and independent of indexing.
   1. An engineer opens a live tail on checkout’s errors containing “timeout”.
   2. The session is registered with every tail worker, indexed by tenant and labels.
   3. Tail workers read new batches from the head of the buffer and match them against their tenant’s sessions.
   4. Matches are pushed to the engineer’s WebSocket within about two seconds, sampled above the session’s rate cap.
   5. Separately, the saved-search evaluator matches every line against 50 k saved searches and counts matches per minute.
   6. The counts are written as metrics, and the monitoring system’s alerting pages the owning team.

Ask: Why not just put every log line into Elasticsearch?

Good answers name: Label index plus compressed chunks in object storage, a token bloom filter per chunk, scan at query time, Inverted index on every field (Elasticsearch / OpenSearch), Columnar store of parsed fields (ClickHouse), Leave logs on the hosts and fan grep out at query time.

Our pick: A stream is a tenant plus a small label set (service, environment, cluster, level, a few more; never ids), capped at about 100 k active streams per tenant. Ingesters build one chunk per stream, flushed at ~4 MB compressed or an hour, each with a bloom filter of its distinct tokens at 1 % false positives (~120 KB, ~3 % of the chunk). The index maps label pairs to streams to chunk refs per tenant per day, ~200 bytes per chunk. Fields are parsed at query time (| json | status >= 500), with only labels and a few promoted fields such as trace_id extracted at write. Grafana Loki is the public example of indexing only labels and scanning chunks in object storage; the bloom filters are what make needle searches for trace ids cheap on top of that. Keep two escape hatches: a small Elasticsearch cluster for security and audit logs, and log-derived metrics or a columnar table for aggregations people run every day.

1. Someone searches for “error” across every service for seven days. What happens?
   With no labels and a common word, the index narrows nothing and every bloom says maybe: that is 7 PB of raw text. The frontend sees this from index stats before running anything. If the search is “latest 100 lines”, newest-first early termination finds 100 matches in the first seconds of data and stops, so it is cheap. If it is a count over the whole week, it is refused with a suggestion (add a service, shorten the range) or offered as a background job against the tenant’s budget; a count people want every day should be a log-derived metric.
2. Engineers want to filter on user_id and trace_id. Why not make them labels?
   Every distinct label set is a stream with its own chunks and index entries. A trace id is unique per request, so 10 M lines a second would become millions of new streams a second, each a tiny chunk: the index would be as big as the data. It is the cardinality problem from the metrics question in a new place. High-cardinality values stay in the line, are found through bloom filters, and can be stored as structured metadata next to each line so they do not need re-parsing.
3. How big are the bloom filters, and why 1 %?
   At 1 % false positives a bloom filter needs about 9.6 bits per distinct token. A 4 MB chunk holds ~40 k lines and ~100 k distinct tokens, so ~120 KB, about 3 % of the chunk. A needle search over 216 k chunks then reads ~26 GB of blooms and ~2,200 false-positive chunks instead of 860 GB. Going to 0.1 % needs 50 % more bloom bytes, ~13 GB more to read, to save about 8 GB of chunk scans, so it does not pay.
4. When would you just run Elasticsearch for everything?
   At a few hundred GB a day. The index overhead is then a few dozen terabytes of SSD, the cluster is a known quantity with excellent tooling, and engineering time is the scarcer resource. The design changes when SSD for the index becomes the largest line item on the bill, which at 1 PB a day it is by an order of magnitude.
5. Schema-on-read sounds slow. Why not parse into fields at write?
   Two thousand services change their log formats with every deploy. Parsing at write means a parse failure either drops data or blocks ingest, and every new field changes the schema. Parsing at read touches only the lines that survive the label and substring filters, usually a tiny fraction. Fields queried constantly (level, status, trace_id) are promoted at write as labels or per-line metadata; everything else stays raw until someone asks.

Ask: The logging pipeline is slow or down. What happens to the services that are logging?

Good answers name: Node agent tails files, checkpoints offsets, buffers on disk, batches with ids, retries, sheds by level, Logging library ships over the network from an in-process buffer, A sidecar agent per pod, Synchronous remote logging (blocking syslog over TCP).

Our pick: Run an agent of the Fluent Bit, Vector or OpenTelemetry Collector class on every host, capped by cgroup at about one core and 512 MB. It tails files, checkpoints (inode, offset) every second only up to what is safely in its disk buffer, follows rotation by inode, stitches stack traces, masks PII and attaches labels. Batches per stream of up to 1 MB or 1 s are compressed with zstd, given an id of agent, stream and sequence, and written to a 1 GB disk buffer (about 28 hours for an average host) before being sent, in order per stream. A batch is deleted from the buffer only after the gateway’s 202, which itself follows Kafka acks=all. On 429 or errors it backs off with jitter; past 50 % of the buffer it drops debug, past 80 % it samples info, and it never drops warn, error or audit. Ingesters drop any batch at or below the highest sequence they have stored for that agent and stream, so the guarantee is at-least-once delivery with effectively-once storage, and every line the agent chooses to drop is counted and reported.

1. The agent itself crashes and restarts. Do you lose or duplicate lines?
   Neither. Batches already formed are in the disk buffer with their ids, so they are re-sent with the same id and deduplicated if the first attempt landed. The file checkpoint only advances to what is safely in that buffer, so lines read but not yet buffered are read again from the file. The cost is a second of re-reading, not a gap.
2. How does the system recognise a retried batch as a duplicate?
   The batch id (agent, stream, sequence) travels into Kafka with the batch, and the partition is a hash of the stream, so the retry lands on the same partition and the same ingester as the original. The agent sends each stream’s batches strictly in order, so the ingester only needs the highest sequence it has stored per (agent, stream) and drops anything at or below it: about a million entries, tens of MB across the fleet. It is the same trick as an idempotent Kafka producer, one level further out.
3. The host’s disk is filling up. What gives way first?
   The agent’s buffer is capped well below free space, and application logs are rotated by size, so logging can never fill the disk the application needs. If the agent cannot keep up and rotation deletes a file it has not finished, those lines are lost and the agent reports a “rotated before read” count for that file. That metric going non-zero is the signal to raise the agent’s cap or cut that service’s volume.
4. Where do you mask PII: the agent or the gateway?
   Both, on purpose. The agent masks first, because a secret that never leaves the host cannot leak in transit, in Kafka or in a backup. The gateway applies the central rule set again, because 100 k agents are never all on the latest version. Card numbers are matched by pattern plus a Luhn check to avoid masking every 16-digit id, and user ids are replaced by a keyed hash so support can still correlate one user’s lines.
5. Why compress on the host rather than at the gateway?
   Bandwidth and money. 10 GB/s raw is 80 Gbit/s crossing the network and often availability zones, where transfer is billed; zstd makes it 8 Gbit/s. An average host compresses 100 KB/s, a rounding error against zstd’s hundreds of megabytes per second per core.

Ask: Thirty days searchable and a year retained, at 100 TB a day compressed. Where does it all live, and how do you delete it?

Good answers name: Time-partitioned chunks per tenant-day: NVMe cache for 3 days, object storage to 30 days, archive class to a year, index and blooms kept in standard storage, retention by prefix delete, Elasticsearch-style index lifecycle: hot nodes, then warm nodes, then delete the index, Everything on replicated SSD for the whole year, Per-line TTLs and delete-by-query.

Our pick: Chunks live at {tenant}/{day}/{hour}/{stream}/{chunk}, and the index is partitioned per tenant per day. Hot is a read-through NVMe cache of about 300 TB on the querier nodes, pre-warmed with each newly flushed chunk and bloom; warm is object storage, 3 PB for 30 days at ~$23 per TB-month; cold is an archive class, ~34 PB at ~$4 per TB-month. The index and bloom filters of cold data (about 3 % of it, ~1 PB) stay in standard storage. The lifecycle job moves tenant-days to the archive at day 30 and deletes the prefix and index partition at the end of the tenant’s retention, skipping legal holds. A cold search becomes a job: plan with the index and blooms, restore only the surviving chunks, scan, and notify when done. Splunk’s indexers use the same idea with hot, warm, cold and frozen buckets.

1. A user asks for their data to be deleted. Logs are immutable. Now what?
   First, the design avoids the problem: user identifiers are masked or hashed at the agent, and anything that must contain them lives under the 30-day retention. For what remains, a weekly batch job finds affected chunks through the blooms on the hashed id, rewrites them without those lines, swaps the chunk refs in the index and deletes the old objects. It is expensive per chunk but touches few chunks, and it runs well inside the legal deadline.
2. Security needs to search six months back, today.
   The index and blooms are still in standard storage, so the planner says within minutes which chunks hold matches and how many bytes would be restored. Expedited restores cost more and finish in minutes, standard ones in hours. If security does this often, keep their own small slice, such as authentication logs at a TB a day, in the warm tier for the full year: 365 TB at ~$23 per TB-month is about $8 k a month.
3. Why three days on SSD, and not one or seven?
   Measure it: plot bytes read by searches against data age, and put the hot boundary where 90 % or more of read bytes are younger. In most companies that is a day or two, and three days of 300 TB costs about $30 k a month. It is a cache, so the boundary can move per tenant without migrating anything.
4. How do you know the archive can actually be restored?
   Restore drills: every week, restore a random sample of chunks from each tenant’s archive and check them against the content hash recorded in the index. A backup that has never been restored is a hope, not a backup. Alert on any mismatch or a restore slower than its target.
5. What do object storage requests cost at this volume?
   About 25 M chunk uploads a day is ~290 PUTs a second, roughly $125 a day at $0.005 per thousand. Reads are cut by the blooms and the hot cache. Small chunks are the enemy of both request cost and query speed, so low-volume streams flush by age and are later compacted into larger chunks.

Ask: A search spans a day and hundreds of thousands of chunks. How does it come back in seconds, and how do you stop one search from eating the cluster?

Good answers name: Frontend splits by hour, prunes with the label index and blooms, scatter-gathers to stateless queriers newest first with early termination, under per-tenant queues and byte budgets, caching finished hours, Send the whole query to every querier and ingester and merge, Precompute the answer as a log-derived metric at ingest.

Our pick: A sub-query is (tenant, hour, selector, filter). The frontend reads index stats to estimate bytes, refuses or defers anything above the tenant’s per-query budget (say 10 TB raw), and dispatches the newest few hours first. Each querier takes a few hundred chunks, fetches blooms and then surviving chunks from the hot cache, decompresses block by block (newest block first for backward searches), runs the substring filter before any parsing, and streams matches. The frontend does a k-way merge by timestamp, streams lines to the user, stops at the limit and cancels the rest. Aggregations such as count_over_time are cached per completed hour, so a dashboard refresh computes only the current hour. Per-tenant queues, concurrency caps and coalescing of identical in-flight sub-queries keep incidents fair, and the querier pool scales on queue wait.

1. Why run the substring filter before parsing?
   A SIMD substring search runs at several GB/s per core; a JSON parser runs at a few hundred MB/s. Filtering first discards 99 % of lines before any parsing, so the planner reorders a query to apply every literal filter it can find before | json, even when the user wrote them the other way round.
2. How do you return lines in time order from thousands of chunks?
   Lines within a stream are stored in ingest order, so each chunk is sorted. A querier merges its chunks, and the frontend merges the queriers’ streams with a heap. For backward searches chunks are made of ~256 KB blocks with min and max timestamps, so a querier decompresses the newest block first and can stop early.
3. A dashboard runs a heavy count over 24 hours every 30 seconds.
   Twenty-three of those hours are immutable, so their counts come from the cache and only the current hour is computed. If it is on many dashboards, turn it into a log-derived metric in the streaming evaluator and the panel stops touching logs at all. Show the cost per panel so the owner sees it too.
4. Hosts’ clocks disagree. How do you order lines across services?
   Lines carry the host’s timestamp, ordered by it, with stream and offset as tie-breakers; the ingest timestamp is kept too. Skew of milliseconds is harmless for reading logs; skew of seconds is a host problem that the agent reports by comparing its clock with the gateway’s. For true causal order across services, follow the trace id, not timestamps.
5. How many queriers, and how do they scale?
   A baseline of ~500 cores covers normal load; incidents bring 5 to 10× the searches for an hour. Queriers hold no data, so they scale on queue wait within a minute or two, from a warm pool. Background and cold-search jobs run on a separate, cheaper pool, so they never compete with someone debugging an outage.

Ask: One team ships a debug flag to production and its log volume goes up twentyfold. How do you protect everyone else, and how do you get teams to log less?

Good answers name: Per-tenant byte and stream quotas at the gateway, shedding by level at the agent, shuffle-sharded partitions, fair query queues, and chargeback on bytes ingested, stored and scanned, One shared capacity sized for the worst case, no quotas, A separate cluster for each large or regulated tenant, Sample everything uniformly under pressure.

Our pick: Quotas are raw bytes a second per tenant and per service, with a 2× burst for 10 minutes, enforced as local token buckets on each gateway node (each holds its share of the quota), plus a cap of about 100 k active streams per tenant against label cardinality. Over quota the gateway answers 429; agents keep the data on disk, drop debug past 50 % of their buffer, keep info at 1 in 10 with a sample_rate label past 80 %, and never drop warn, error or audit. Each tenant is shuffle-sharded onto 8 of the 512 partitions (more for large tenants), and ingesters cap streams and memory per tenant. Queries queue per tenant with concurrency caps and byte budgets. Usage is metered per tenant, service and level: bytes ingested, GB-days per tier and bytes scanned, priced and charged back monthly, with a report of each team’s ten most expensive log statements, found by grouping lines into templates with a template miner such as Drain.

1. In the middle of their incident, the team needs exactly the debug logs you are dropping.
   Give them a self-serve, time-boxed quota raise (say 4× for an hour), charged to their budget and visible to their manager. Better still is debug on demand: raise the level on a few pods for 15 minutes, tagged to a short-retention class of 3 days. The shedding order is published in advance, so nobody is surprised by what goes first.
2. How much does shuffle sharding actually help?
   With 8 of 512 partitions per tenant, another random tenant overlaps on at least one partition about 12 % of the time and on all 8 essentially never. So a flood degrades at most one or two of a neighbour’s partitions, and its other partitions keep flowing. Without it, a flood on shared partitions hits everyone on them.
3. Sampling breaks counts. How do teams still get correct error rates?
   Errors and warnings are never sampled. For sampled info lines the stored sample_rate lets count queries scale back, which gives good estimates for frequent lines. Exact counts of frequent events belong in metrics anyway; the agent can emit per-pattern counts before sampling, so the totals are exact even when the lines are not all kept.
4. How do you find the log statements that cost the most?
   Sample about 1 % of lines at ingest, group them into templates by masking numbers, ids and quoted values (Drain does this well), and sum bytes per template per service. A report saying “this line in checkout costs $9 k a month and nobody searched it in 90 days” cuts volume faster than any quota.
5. One team’s searches scan 20 TB a day. Is that a problem?
   Only if it is unbounded or unfair. Bytes scanned are metered and charged like storage, per-query budgets stop accidents, and the frontend’s per-tenant queues keep it from hurting others. Usually the fix is a label they forgot or a log-derived metric for the question they keep asking.

Ask: Why not implement live tail as a search for the last two seconds, repeated every second?

Good answers name: Separate consumer groups on the buffer: tail workers match active sessions from the head; an evaluator matches saved searches and emits counts as metrics, Poll the search API every second, Tail from the ingesters’ in-memory head chunks, Stream straight from the agents to the browser.

Our pick: A tail session is (tenant, selector, line filter, rate cap) and lives on the node holding the engineer’s WebSocket. The session registry is small, a few thousand entries, so every tail worker holds all of it, indexed by tenant and labels. Workers form the "tail" consumer group, start at the latest offset and skip ahead if they fall more than ~10 s behind; they match labels per batch and the filter per line, and forward matches to the session’s node, which caps delivery at 500 lines a second and reports skipped lines. Saved searches work the same way in an "alerts" group: ~50 k searches indexed by tenant and labels, literal filters compiled into one multi-pattern matcher per tenant, counts per search per minute emitted as metrics to the monitoring system, whose alerting already does for-durations, grouping, routing and a dead man’s switch. Searches that parse and group over long windows run as scheduled queries with cached hours.

1. A tail session’s selector matches 2 GB/s.
   Matching costs the same whatever the session matches, because every line is examined anyway; what grows is forwarding. The cap bounds that at 500 lines a second per session, the tail samples beyond it, and the UI shows “48,000 lines a second skipped, add a filter”. Nobody can read 2 GB/s, so this loses nothing a human would have seen.
2. The evaluator restarts. Do alert counts go wrong?
   It checkpoints its counters together with its offsets, so a restart resumes from a consistent point and recounts only the lines after it. It emits running totals per search rather than increments, so re-sending a value after the restart overwrites instead of adding. Alert rules hold for minutes, so the few seconds of delay never change a page.
3. Lines buffered on a host for an hour arrive now. Do they fire old alerts?
   Alert counts use arrival time, because an alert is about what is happening now, and lines older than about 10 minutes on arrival are left out of alert counts. They are still indexed under their real timestamps, so searches and dashboards show them in the right place.
4. Why send counts to the metrics system instead of paging directly?
   Paging well is a product of its own: for-durations, grouping, silences, inhibition, routing by owner, escalation and a check that the paging path itself is alive. The metrics stack already has all of it, and on-call teams already write their rules there. Log-derived metrics also give cheap dashboards without touching the logs.