Observability, operations and rollouts

Metrics, logs and traces; SLIs, SLOs and error budgets; alerting; safe deploys (canary, blue-green, feature flags); migrations without downtime; capacity planning; and how to answer "how would you know it is working".

A design that cannot be observed, deployed safely, or migrated is not finished, and staff-level interviews check for it. The questions are phrased "how would you know it is working", "how do you roll this out", and "how do you migrate from the old system". Each has a short standard answer that you should be able to give for any component in your diagram.

The three signals

Metrics are numbers over time, aggregated: request rate, error rate, latency percentiles, queue depth, cache hit rate, replication lag. Cheap to store, fast to query, the basis of dashboards and alerts. Emit them per service and per dependency with labels for endpoint and status. Prometheus, Datadog, CloudWatch.

Logs are per-event records with context. Structured (JSON with request id, user id, latency, outcome), sampled at high volume, and searchable. The place you go after a metric alerts.

Traces follow one request through every service it touches, with a span per hop and timing. They answer "which of the seven calls made this request slow". Propagate a trace id in headers (W3C traceparent); OpenTelemetry is the standard; Jaeger, Tempo, Honeycomb store them. Sample 1 to 10 % of traffic plus every error.

The interview answer: "Every service emits RED metrics (rate, errors, duration) per endpoint and per downstream, structured logs with the trace id, and OpenTelemetry traces sampled at 5 %."

SLIs, SLOs and error budgets

An SLI is a measured indicator: the fraction of requests that succeeded in under 300 ms. An SLO is the target: 99.9 % over 30 days. The error budget is what is left: 0.1 % of requests, or 43 minutes of full outage per month. Alerts fire on the rate at which the budget is burning (a fast burn pages, a slow burn opens a ticket), not on individual thresholds, which cuts noise. When the budget is spent, the team stops feature work and fixes reliability. Google's SRE model; saying "SLO on the checkout path of 99.95 % success within 500 ms, alert on burn rate" is a complete answer.

Choose SLIs for what users experience: end-to-end latency at the gateway, message delivery time from send to receive, feed freshness, not CPU or memory.

What to alert on

Symptoms, not causes: error rate, latency, and staleness at the user-facing edge. Add a few leading indicators that predict user impact before it happens: consumer lag, replication lag, queue age, disk, certificate expiry, and saturation of the pressured resource you identified in estimation. Every alert should be actionable and have a runbook; an alert nobody acts on gets deleted.

Health checks and dependencies

Expose a shallow health endpoint for the load balancer and a deep one for dashboards. Instrument every dependency call with its own latency and error metrics so you can see which dependency is failing before the trace tells you. Track the circuit breaker state as a metric.

Safe deploys

Deploys cause most incidents, so the rollout is part of the design.

• Rolling. Replace instances a few at a time behind the load balancer with connection draining. Simple; a bad version affects a growing share of traffic until you notice.
• Canary. Send 1 % of traffic to the new version, compare its error rate and latency to the baseline automatically, and proceed to 10 %, 50 %, 100 % only if they match. Abort and roll back on regression. The default for large services.
• Blue-green. Run two full environments and switch the router. Instant rollback; double the capacity during the switch; database changes must be compatible with both.
• Feature flags. Ship code dark, enable per user, per cohort, per region, and turn off in seconds without a deploy. Decouples deploy from release and is how risky changes are tested in production safely.

Always: automated rollback triggered by SLO regression; every change (code, config, flags, schema) is a deploy with the same controls; nothing is rolled out globally at once, including config.

Schema migrations without downtime

Rule: every migration must work with both the old and the new code running at once, because they will be. The expand and contract pattern:

1. Expand. Add the new column, table or index (nullable, no default that rewrites the table). Deploy code that writes both old and new.
2. Backfill. Copy old data into the new shape in batches, rate limited, resumable.
3. Switch reads to the new shape behind a flag; verify with a comparison job.
4. Contract. Stop writing the old shape, then drop it in a later release.

Large-table operations (adding an index, changing a type) use online tools (CREATE INDEX CONCURRENTLY, gh-ost, pt-online-schema-change) so they do not lock the table.

Migrating between systems

Moving from one database or service to another follows the same shape at a larger scale: dual write (new writes go to both), backfill the old data, shadow read (read from both, compare, serve from old), cut over reads with a flag and the ability to flip back, then stop the old writes and decommission. Weeks to months for anything large; the comparison step is what catches the bugs. Saying this sequence unprompted is a strong signal.

Capacity planning

From the estimation: the pressured resource, its per-unit capacity, and a headroom factor (run at 50 to 60 % utilisation so a zone loss or a spike does not saturate). Autoscale stateless tiers on CPU or request queue length; scale stateful tiers ahead of demand, because adding a database node takes hours to rebalance. Load test the pressured path before launch at 2× projected peak and know where it breaks.

Cost

A staff-level touch: name the expensive part. Egress and video bytes, cross-region replication, always-on cache memory, log volume at 100 % sampling. Say what you would trade: shorter retention, sampling, a colder storage tier, compressing before storing.

Incident readiness

Runbooks for each alert; a regular game day where you kill a dependency and watch the degradation work; backups tested by restoring them; a documented failover procedure with its expected RTO (time to recover) and RPO (data loss window). "We restore from backup" is not an answer until it has been done.

In the interview

"I would know it is working from the delivery-latency SLI: time from send to receive, p99 under 500 ms, SLO 99.9 %, alerting on burn rate. Consumer lag on the fan-out topic is the leading indicator. Rollouts are canary with automated rollback; the schema change for the new index is expand-and-contract with a concurrent index build. The migration from the old store dual-writes for two weeks with shadow reads before cutover." That is the whole operations section in four sentences.