Health checks, outlier detection and the black hole effect: an interactive simulator

A load balancer has to stop sending traffic to servers that cannot serve it. Active health checks probe each server on a schedule; passive health checks, or outlier detection, watch the responses to real requests. The trap is a server that fails fast: it looks healthy to a shallow probe, and to a least-connections balancer it looks like the least busy server, so it attracts most of the traffic. The effect is called a black hole.

The playground sends a minute of traffic to five servers; at 10 seconds one of them loses a dependency and starts failing every request in 2 ms. Compare four balancer setups and two kinds of health check.

Why fast failures attract traffic

Least connections sends each request to the server with the fewest requests in progress, which is usually smart: it steers around slow servers. But a server that answers every request with an error in 2 ms, where healthy servers take 50 ms, always has the fewest requests in progress. In the playground it ends up with 78% of all traffic, so a fault in one server of five fails most requests. Round robin, blind as it is, caps the damage at a fifth.

Checks that see the failure

• Shallow checks that return 200 if the process is running keep passing while every real request fails. They are right for liveness, deciding whether to restart a wedged process, but not for routing.
• Deep checks exercise what a request needs: a database connection, a downstream call, a cache. Here the deep check takes the server out after three failures five seconds apart, at 20 s; until then, errors pile up. Keep them cheap and do not let a shared dependency’s outage fail every server’s check at once.
• Outlier detection needs no probe: five errors in a row from one server eject it within milliseconds, and it is retried after a back-off. Cap how much of the fleet can be ejected, so a bad deploy or a global dependency failure does not empty the pool.
• Count errors as load. Some balancers weight least-connections or latency-based choices by error rate, so a failing server stops looking attractive.

Outlier detection and probes in configuration

# Envoy: eject a host after 5 consecutive 5xx responses, for 30 s at first
outlier_detection:
  consecutive_5xx: 5
  interval: 1s
  base_ejection_time: 30s            # grows each time the same host is ejected again
  max_ejection_percent: 20           # never eject more than a fifth of the fleet

# Kubernetes: keep liveness shallow, make readiness reflect real ability to serve
livenessProbe:                        # restart the container only if the process is wedged
  httpGet: { path: /livez, port: 8080 }
readinessProbe:                       # take it out of the load balancer if it cannot do work
  httpGet: { path: /readyz, port: 8080 }   # checks the database pool and key dependencies
  periodSeconds: 5
  failureThreshold: 3

In a system design interview

When you put a load balancer in a design, say how it learns a backend is bad: “Readiness checks exercise the database pool every five seconds; liveness only checks the process. The mesh also runs outlier detection, ejecting a host after five consecutive 5xx for 30 seconds, never more than 20% of hosts at once. Least-request balancing is weighted by error rate, so a host that fails fast is not mistaken for an idle one.” Expect the follow-ups: what happens when every backend fails its check, and how retries interact with ejection.

Read the guide to load balancing or service discovery and service mesh, see load balancing algorithms and the circuit breaker, or watch a strong candidate design a URL shortener.

Questions people ask