Updated 24 October 2026

Observability, operations and rollouts

Metrics, logs and traces; SLIs, SLOs and error budgets; alerting; safe deploys (canary, blue-green, feature flags); migrations without downtime; capacity planning; and how to answer "how would you know it is working".

Try it yourself, in a playground:Canary deploymentsBurn rate alertingTrace samplingTime-series rollupsHealth checksReservoir samplingCoordinated omission

A design that cannot be observed, deployed safely, or migrated is not finished, and staff-level interviews check for it. The questions are phrased "how would you know it is working", "how do you roll this out", and "how do you migrate from the old system". Each has a short standard answer that you should be able to give for any component in your diagram.

The three signals

Metrics are numbers over time, aggregated: request rate, error rate, latency percentiles, queue depth, cache hit rate, replication lag. Cheap to store, fast to query, the basis of dashboards and alerts. Emit them per service and per dependency with labels for endpoint and status. Prometheus, Datadog, CloudWatch.

Logs are per-event records with context. Structured (JSON with request id, user id, latency, outcome), sampled at high volume, and searchable. The place you go after a metric alerts.

Traces follow one request through every service it touches, with a span per hop and timing. They answer "which of the seven calls made this request slow". Propagate a trace id in headers (W3C traceparent); OpenTelemetry is the standard; Jaeger, Tempo, Honeycomb store them. Sample 1 to 10 % of traffic plus every error.

The interview answer: "Every service emits RED metrics (rate, errors, duration) per endpoint and per downstream, structured logs with the trace id, and OpenTelemetry traces sampled at 5 %."

SLIs, SLOs and error budgets

An SLI is a measured indicator: the fraction of requests that succeeded in under 300 ms. An SLO is the target: 99.9 % over 30 days. The error budget is what is left: 0.1 % of requests, or 43 minutes of full outage per month. Alerts fire on the rate at which the budget is burning (a fast burn pages, a slow burn opens a ticket), not on individual thresholds, which cuts noise. When the budget is spent, the team stops feature work and fixes reliability. Google's SRE model; saying "SLO on the checkout path of 99.95 % success within 500 ms, alert on burn rate" is a complete answer.

Choose SLIs for what users experience: end-to-end latency at the gateway, message delivery time from send to receive, feed freshness, not CPU or memory.

What to alert on

Symptoms, not causes: error rate, latency, and staleness at the user-facing edge. Add a few leading indicators that predict user impact before it happens: consumer lag, replication lag, queue age, disk, certificate expiry, and saturation of the pressured resource you identified in estimation. Every alert should be actionable and have a runbook; an alert nobody acts on gets deleted.

Health checks and dependencies

Expose a shallow health endpoint for the load balancer and a deep one for dashboards. Instrument every dependency call with its own latency and error metrics so you can see which dependency is failing before the trace tells you. Track the circuit breaker state as a metric.

Safe deploys

Deploys cause most incidents, so the rollout is part of the design.

• Rolling. Replace instances a few at a time behind the load balancer with connection draining. Simple; a bad version affects a growing share of traffic until you notice.
• Canary. Send 1 % of traffic to the new version, compare its error rate and latency to the baseline automatically, and proceed to 10 %, 50 %, 100 % only if they match. Abort and roll back on regression. The default for large services.
• Blue-green. Run two full environments and switch the router. Instant rollback; double the capacity during the switch; database changes must be compatible with both.
• Feature flags. Ship code dark, enable per user, per cohort, per region, and turn off in seconds without a deploy. Decouples deploy from release and is how risky changes are tested in production safely.

Always: automated rollback triggered by SLO regression; every change (code, config, flags, schema) is a deploy with the same controls; nothing is rolled out globally at once, including config.

Schema migrations without downtime

Rule: every migration must work with both the old and the new code running at once, because they will be. The expand and contract pattern:

1. Expand. Add the new column, table or index (nullable, no default that rewrites the table). Deploy code that writes both old and new.
2. Backfill. Copy old data into the new shape in batches, rate limited, resumable.
3. Switch reads to the new shape behind a flag; verify with a comparison job.
4. Contract. Stop writing the old shape, then drop it in a later release.

Large-table operations (adding an index, changing a type) use online tools (CREATE INDEX CONCURRENTLY, gh-ost, pt-online-schema-change) so they do not lock the table.

Migrating between systems

Moving from one database or service to another follows the same shape at a larger scale: dual write (new writes go to both), backfill the old data, shadow read (read from both, compare, serve from old), cut over reads with a flag and the ability to flip back, then stop the old writes and decommission. Weeks to months for anything large; the comparison step is what catches the bugs. Saying this sequence unprompted is a strong signal.

Capacity planning

From the estimation: the pressured resource, its per-unit capacity, and a headroom factor (run at 50 to 60 % utilisation so a zone loss or a spike does not saturate). Autoscale stateless tiers on CPU or request queue length; scale stateful tiers ahead of demand, because adding a database node takes hours to rebalance. Load test the pressured path before launch at 2× projected peak and know where it breaks.

Cost

A staff-level touch: name the expensive part. Egress and video bytes, cross-region replication, always-on cache memory, log volume at 100 % sampling. Say what you would trade: shorter retention, sampling, a colder storage tier, compressing before storing.

Incident readiness

Runbooks for each alert; a regular game day where you kill a dependency and watch the degradation work; backups tested by restoring them; a documented failover procedure with its expected RTO (time to recover) and RPO (data loss window). "We restore from backup" is not an answer until it has been done.

In the interview

"I would know it is working from the delivery-latency SLI: time from send to receive, p99 under 500 ms, SLO 99.9 %, alerting on burn rate. Consumer lag on the fan-out topic is the leading indicator. Rollouts are canary with automated rollback; the schema change for the new index is expand-and-contract with a concurrent index build. The migration from the old store dual-writes for two weeks with shadow reads before cutover." That is the whole operations section in four sentences.

Cardinality, sampling and cost

Observability bills grow faster than traffic if nobody is watching.

• Metrics cost by unique time series: labels × values. A label with a user id or a request id multiplies your series count by millions and is the usual cause of a surprise bill or a dead Prometheus. Keep labels to bounded sets: service, endpoint, region, status class.
• Logs cost by volume. Log one structured line per request at info, with the fields you would actually filter on (trace id, user id, endpoint, status, duration), and put the details behind debug that can be raised for one service temporarily.
• Traces cost by span. Head-based sampling at 1–10 % for normal traffic, plus tail-based sampling that keeps everything for errors and slow requests. Always propagate the trace id everywhere, even for unsampled requests, so a log line can be joined to a trace.

The rule of thumb to say: metrics for what, traces for where, logs for why. Reaching for logs first is what makes incidents long.

Runbooks and on-call

Every alert needs a runbook entry with four things: what fired and what it means for users, the first three checks, the mitigation (usually roll back, fail over, or turn a flag off), and who to escalate to. An alert without a runbook trains people to ignore alerts.

Say what makes on-call sustainable: alerts that are actionable and rare, dashboards per service that answer "is it us or a dependency", a documented rollback that takes minutes, and a blameless review after anything user-visible with action items that have owners and dates.

Feature flags as an operational tool

Flags are not just for product experiments. They are the fastest mitigation available: turn off the expensive new query, drop to the degraded path, disable a noisy integration. That makes flags production configuration, which in turn means they need evaluation without a redeploy, a safe default when the flag service is unreachable, an audit trail of who flipped what, and cleanup: a flag that has been on for a year is dead code with a branch in it.

Checklist

• The three or four SLIs that reflect the user experience, with targets.
• What pages a human, and why each of those is actionable.
• Trace propagation and the sampling policy.
• Deploy strategy, with the rollback time and its trigger.
• The migration procedure for the biggest schema change in the design.
• Where the cost goes, and the one lever that reduces it.

Questions people ask

What is observability in system design?

Observability is being able to answer "how would you know it is working" for every component in a design, using three signals: metrics, logs and traces. A complete answer: every service emits RED metrics (rate, errors, duration) per endpoint and per downstream, structured logs with the trace id, and OpenTelemetry traces sampled at 5 %. Staff-level interviews check for it, because a design that cannot be observed is not finished.

Metrics vs logs vs traces: what is the difference?

Metrics are aggregated numbers over time, such as request rate, error rate and latency percentiles, cheap to store and the basis of dashboards and alerts. Logs are structured per-event records with context, the place to go after a metric alerts. Traces follow one request through every service it touches, with a span per hop, and answer which call made a request slow. The rule of thumb: metrics for what, traces for where, logs for why.

SLI vs SLO vs error budget: what is the difference?

An SLI is a measured indicator, such as the fraction of requests that succeeded in under 300 ms; an SLO is the target, such as 99.9 % over 30 days; and the error budget is what is left, here 0.1 % of requests or 43 minutes of full outage per month. Alerts fire on the rate at which the budget is burning, which cuts noise. When the budget is spent, the team stops feature work and fixes reliability.

What should you alert on?

Alert on symptoms, not causes: error rate, latency and staleness at the user-facing edge. Add a few leading indicators that predict user impact, such as consumer lag, replication lag, queue age, disk, certificate expiry and saturation of the pressured resource. Every alert should be actionable and have a runbook, because an alert without one trains people to ignore alerts.

How do you run a schema migration without downtime?

Use the expand and contract pattern, because old and new code will run at once and every migration must work with both. Expand by adding the new column, table or index and writing both shapes; backfill old data in rate-limited, resumable batches; switch reads behind a flag and verify with a comparison job; then stop writing the old shape and drop it in a later release. Large-table operations use online tools such as CREATE INDEX CONCURRENTLY.

How do you migrate from one database to another?

Dual write so new writes go to both systems, backfill the old data, shadow read from both and compare while still serving from the old one, cut over reads with a flag that can flip back, then stop the old writes and decommission. For anything large this takes weeks to months. The comparison step is what catches the bugs.