Head vs tail sampling for distributed tracing: an interactive simulator

Distributed tracing records every hop of a request as spans, which adds up fast: at 10,000 requests a second with 30 spans each, keeping everything is about 26 TB a day. So tracing systems sample. Head sampling decides when a trace starts, cheaply but blind to how it turns out. Tail sampling decides when it ends, so it can keep the traces worth looking at: the failures and the slow ones.

The playground plots 4,000 requests by time and latency, with failures in red, and shows which traces each strategy keeps. The table compares them on the same requests.

Head sampling: cheap and blind

The first service a request reaches flips a coin and passes the decision along in the trace context, the sampled flag of the W3C traceparent header, so every service records the trace or none does. It costs nothing to decide. But it keeps failures at the same rate as everything else, and failures are rare: with 1% of requests failing, sampling 1% keeps 47 traces, of which 1 is a failure and none are slow. Of the 5 failures on the rare checkout endpoint, where the actual incident is, it keeps none, and even sampling 10% keeps none of them.

Tail sampling: keep what matters

A collector buffers every span, grouped by trace id, until the trace is complete, then applies rules: keep every error, every trace over a latency threshold, and a small random share of the rest as a baseline. In the playground that keeps 136 traces, 3.4% of the total, and every one of the 45 failures, 45 slow requests and 5 checkout failures. It has costs of its own:

• Memory. Every span of every unfinished trace is held while waiting to decide: about 9 GB across the collectors at 10,000 requests a second and a 30-second wait.
• Routing. All of a trace’s spans must reach the same collector, so a first tier routes spans by trace id to a second tier that decides.
• Late spans. A span that arrives after the decision belongs to a trace already kept or dropped; asynchronous work makes “finished” fuzzy.

Many teams combine them: a head-sampling rate high enough to keep load manageable, tail sampling on top, and metrics for the counts. A sampled trace should record its sampling rate, so that anything counted from traces can be scaled back up.

Head and tail sampling with OpenTelemetry

# head sampling in every service's OpenTelemetry SDK: keep 1% of new traces,
# and follow the caller's decision for the rest
#   OTEL_TRACES_SAMPLER=parentbased_traceidratio
#   OTEL_TRACES_SAMPLER_ARG=0.01

# tail sampling in the OpenTelemetry Collector
processors:
  tail_sampling:
    decision_wait: 30s              # hold spans until the trace is complete
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow
        type: latency
        latency: { threshold_ms: 1000 }
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 1 }

In a system design interview

When observability comes up, separate the signals: “Metrics tell us something is wrong, from every request. Traces tell us where, so we keep a sample: services propagate W3C trace context, send all spans to collectors routed by trace id, and tail sampling keeps every error and every request over the latency SLO, plus 1% of the rest. That keeps every trace an on-call engineer is likely to open at a few percent of the storage.” Expect the follow-ups: the cost of the collector tier, linking logs to traces, and sampling for very high-volume endpoints.

Read the guide to distributed tracing or observability, see SLO burn rate alerts for deciding when to look, or watch a strong candidate design a monitoring system.

Questions people ask