Configuring Tail Sampling in the Collector

Tail sampling buffers every span of a trace until the trace is complete, then applies policies to the whole thing — so it can keep 100% of errors and slow traces while dropping most successful ones, at the cost of holding several gigabytes of spans in memory and requiring that every span of a trace reach the same Collector instance.

Context and when it matters

Head-based sampling decides at the root, before anything is known. That is cheap and predictable, and it means a rare error is kept only by luck — at a 1% rate you see one error trace in a hundred, and the one you need is almost certainly among the ninety-nine that were dropped. See parent-based vs always-on sampler behaviour for why raising a downstream rate does not help.

Tail sampling inverts the trade. The decision moves to the Collector and happens after the outcome is known, so “keep every trace that errored, every trace over two seconds, and 2% of the rest” becomes expressible. What you pay is memory, latency to storage, and a hard architectural constraint: the deciding Collector must see the entire trace.

The architectural constraint

Why tail sampling needs trace-ID-aware routing Services send spans to agent collectors. The agents forward to a load balancing exporter layer that hashes on trace ID. Each tail sampling collector therefore receives every span belonging to the traces assigned to it, which is required for the sampling decision to see the complete trace. Round-robin routing silently breaks tail sampling service A service B service C agent layer loadbalancing exporter routing key hash(traceID) sampler 1 traces a–m sampler 2 traces n–z sampler 3 traces 0–9 If two samplers each see half a trace, both judge an incomplete trace — and an error span may land on the instance that never saw it. 
# agent layer — route by trace ID so each sampler sees whole traces
exporters:
  loadbalancing:
    routing_key: traceID          # NOT service — traceID is the requirement
    protocol:
      otlp:
        tls: { insecure: true }
    resolver:
      k8s:
        service: otel-tailsampler.observability

The policy set

Policies are evaluated per trace; a trace is kept if any policy says so. Order them by value, and always finish with a probabilistic catch-all so you retain a baseline of healthy traffic for comparison.

processors:
  tail_sampling:
    # Must exceed the p99.9 duration of your slowest trace, plus export lag.
    decision_wait: 30s
    num_traces: 100000            # traces held in memory at once
    expected_new_traces_per_sec: 2000
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }

      - name: keep-slow
        type: latency
        latency: { threshold_ms: 2000 }

      # Business-critical paths worth full retention regardless of outcome.
      - name: keep-payments
        type: string_attribute
        string_attribute:
          key: service.name
          values: [payments-api]

      # Debug override: an authenticated header forced this trace to be kept.
      - name: keep-forced
        type: boolean_attribute
        boolean_attribute: { key: sampling.force_keep, value: true }

      # Baseline of successful traffic, so latency comparisons stay honest.
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 2 }

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp/tempo]

The baseline policy is the one people omit and later regret. Without a sample of successful traces there is nothing to compare an error trace against, and every latency percentile computed from stored traces is biased towards failures.

Composite and rate-limiting policies

For finer control, a composite policy applies sub-policies with individual rate limits, which prevents one noisy service from consuming the entire budget:

      - name: composite-budget
        type: composite
        composite:
          max_total_spans_per_second: 5000
          policy_order: [errors-first, slow-second, everything-else]
          composite_sub_policy:
            - name: errors-first
              type: status_code
              status_code: { status_codes: [ERROR] }
            - name: slow-second
              type: latency
              latency: { threshold_ms: 1500 }
            - name: everything-else
              type: always_sample
          rate_allocation:
            - policy: errors-first
              percent: 60
            - policy: slow-second
              percent: 30
            - policy: everything-else
              percent: 10

Memory is the real constraint

Every span of every in-flight trace is held until the decision. The arithmetic is unforgiving:

Tail sampling buffer arithmetic Two thousand traces per second multiplied by a thirty second decision wait gives sixty thousand traces in flight. At sixty spans per trace and eight hundred bytes per span in memory, that is about 2.9 gigabytes of buffer before overhead. Buffer = rate × wait × spans × bytes 2,000 traces/s post head-sampling × 30 s decision wait > slowest trace × 60 spans/trace measured × 800 B/span in memory ≈ 2.9 GB of buffer per sampler instance, before Go runtime overhead Halving decision_wait halves the buffer — and drops every trace slower than the new window. In-memory spans are roughly twice their compressed wire size; measure rather than assuming.

decision_wait is the parameter with the sharpest trade-off. Too short and slow traces — precisely the ones the latency policy exists to capture — are judged incomplete and dropped. Too long and memory grows linearly. Set it above the p99.9 of end-to-end trace duration, then leave it alone.

Designing the policy set

Policies look simple and accumulate badly. A set that starts as “keep errors and slow traces” becomes a dozen overlapping rules that nobody can reason about, and because a trace is kept if any policy matches, the effective retention rate drifts upward until storage costs force a review.

Three habits keep the set manageable.

Write down what each policy is for. Not what it matches — that is in the configuration — but which question it exists to answer. “Keep-payments exists so the finance team can audit any transaction” is a rationale that can be revisited; “string_attribute service.name = payments-api” is not. Policies without a stated purpose are never removed, because nobody can argue against them.

Measure each policy’s contribution. The Collector exposes sampled counts per policy, so you can see which ones are actually doing work. A policy matching nothing is dead configuration; a policy matching forty percent of traffic is your real sampling rate wearing a disguise. Both deserve attention, and both are invisible without the per-policy metric.

Prefer few broad policies to many narrow ones. Every additional policy multiplies the reasoning required to predict what is kept, and narrow policies tend to encode a specific incident rather than a durable need. A set of five — errors, slow, one or two business-critical paths, a debug override, and a probabilistic baseline — covers nearly every requirement teams actually have.

One further consideration: policies interact with the span attributes your services emit. A policy keyed on an attribute that only some services set will behave inconsistently across the fleet, and the inconsistency will look like a sampling bug rather than an instrumentation gap. When adding a policy, confirm the attribute it depends on is set everywhere it needs to be.

Verification

# Decisions by policy — every policy should be firing
sum by (policy) (rate(otelcol_processor_tail_sampling_count_traces_sampled[5m]))

# Traces dropped because the buffer was full — must be zero
rate(otelcol_processor_tail_sampling_sampling_trace_dropped_too_early[5m])

# Actual buffer occupancy against num_traces
otelcol_processor_tail_sampling_sampling_trace_status_count

Then confirm the behaviour end to end: force an error on a test endpoint and check the trace is stored despite a 2% baseline, then issue a normal request and confirm it usually is not.

Traces kept per policy, one hour Traces kept per policy, one hour. errors 7,200 · 100% of errors; slow (over 2 s) 4,600; payments path 3,100; probabilistic baseline 2% 23,000 Traces kept per policy, one hour errors 7,200 · 100% of errors slow (over 2 s) 4,600 payments path 3,100 probabilistic baseline 2% 23,000 The baseline dominates the count and is what makes the other three interpretable.

Common pitfalls

  • Round-robin load balancing. The most common misconfiguration; each sampler sees a fragment and judges it, so error traces are kept partially or not at all.
  • decision_wait shorter than the slowest trace. Silently drops exactly the traces the latency policy targets.
  • No baseline policy. Storage fills with failures only, and every comparison against “normal” becomes impossible.
  • Running tail sampling on the agent layer. An agent sees only its own node’s spans. Tail sampling belongs in a gateway tier — see deploying the Collector as agent vs gateway.
  • Forgetting head sampling still applies. If the SDK dropped the trace, the Collector never sees it. Tail sampling can only select from what was exported.

The latency cost nobody mentions

Tail sampling delays every trace by the decision wait. With a thirty-second window, a trace is not queryable until roughly thirty seconds after the request completed, which changes how the system feels during an incident: you fix something, reload the trace view, and the evidence is not there yet.

That is usually acceptable and occasionally not. If your incident workflow depends on seeing a trace seconds after reproducing a problem, either shorten the window and accept dropping the slowest traces, or run a parallel head-sampled pipeline at a low rate that writes immediately. The second option costs a little storage and removes the wait entirely for the traces you are actively watching.

Finally, remember that tail sampling composes with head sampling rather than replacing it. Most production pipelines run a modest head rate to bound Collector ingest, then tail sampling to decide what is worth storing from what survived — the two stages answer different questions and tuning them together is what keeps both memory and storage predictable.

Related

↑ Back to OpenTelemetry Collector Pipeline Configuration