Parent-Based vs Always-On Sampler Behaviour

ParentBased obeys the sampled flag on the incoming traceparent and only consults its own sampler when there is no parent — so a rate change on a downstream service changes nothing, because by the time a request reaches it the decision has already been made at the trace root.

Context and when it matters

The classic version of this confusion: a team notices the payments service produces far more spans than expected, sets OTEL_TRACES_SAMPLER_ARG=0.01, redeploys, and sees no change at all. Nothing is broken. The payments service is a downstream service, every request arriving at it already carries a sampled traceparent, and ParentBased — the default in every SDK — respects that decision without consulting the ratio at all.

Understanding the delegation model matters for three practical reasons: it tells you where a rate change is effective (only at trace roots), it explains why some services see 100% of traffic no matter what you configure, and it is the mechanism behind broken traces where half the spans are stored and half are not.

How ParentBased actually decides

ParentBased is not itself a sampling algorithm. It is a router with five branches, four of which are usually left at their defaults.

The five branches of ParentBased A decision tree. If there is no parent span context, the root sampler decides, typically a trace ID ratio. If the parent is remote and sampled, the span is recorded. If the parent is remote and not sampled, the span is dropped. If the parent is local and sampled, the span is recorded. If the parent is local and not sampled, the span is dropped. Only the first branch consults a configured ratio. Only the top branch reads your configured rate new span what is the parent? no parent (trace root) → root sampler decides remote parent, sampled remote parent, not sampled local parent, sampled local parent, not sampled your rate applies here traceidratio 0.01 RECORD_AND_SAMPLE DROP RECORD_AND_SAMPLE DROP

The four parent branches default to “do what the parent did”, and that default is what makes traces coherent: either the whole trace is stored or none of it is. Overriding them is possible and almost always a mistake, because a trace with holes is harder to read than no trace at all.

The samplers you can choose from

# The five standard values of OTEL_TRACES_SAMPLER
OTEL_TRACES_SAMPLER=always_on                  # keep everything, no delegation
OTEL_TRACES_SAMPLER=always_off                 # keep nothing
OTEL_TRACES_SAMPLER=traceidratio               # keep a ratio, IGNORING the parent
OTEL_TRACES_SAMPLER=parentbased_always_on      # follow parent, keep roots  ← common default
OTEL_TRACES_SAMPLER=parentbased_traceidratio   # follow parent, ratio at roots
OTEL_TRACES_SAMPLER_ARG=0.05                   # the ratio, when one applies
The five standard samplers at a glance Five rows. always_on ignores the parent and keeps everything, used in development. always_off ignores the parent and keeps nothing, used to disable tracing. traceidratio ignores the parent and applies a ratio, used only when every service shares the same ratio. parentbased_always_on obeys the parent and keeps all roots, the common default. parentbased_traceidratio obeys the parent and applies a ratio at roots, the production choice. Obeys the parent? · What happens at a root? sampler obeys parent at a root use it for always_on no keep all local development always_off no drop all disabling one service traceidratio no apply ratio uniform fleet only parentbased_always_on yes keep all roots the SDK default parentbased_traceidratio yes ratio at roots production

The two that surprise people:

parentbased_always_on — the SDK default in most languages — keeps 100% of traces that originate in this service. On an edge service that is every request. Teams frequently discover their “1% sampling” only applies to a downstream service that never starts a trace, while the gateway is running at 100%.

traceidratio without the parentbased_ prefix ignores the parent entirely and decides independently. Because the decision is derived from the trace ID, two services with the same ratio and the same trace ID reach the same conclusion — which is why this produces coherent traces despite not consulting the parent, and why two services with different ratios produce traces with holes.

What actually happens across a call chain

# Reproduce it locally: three services, three configurations.
#
#   gateway         parentbased_traceidratio, arg=0.10   → decides for the trace
#   checkout        parentbased_traceidratio, arg=0.01   → obeys gateway, arg unused
#   payments        parentbased_always_on                → obeys gateway
#
# Result: 10% of traces stored, complete. Changing checkout's 0.01 to 0.5
# changes nothing at all, because checkout never sees a request without a parent.

from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased

# Explicit construction, showing what the environment variable builds:
sampler = ParentBased(
    root=TraceIdRatioBased(0.10),        # consulted ONLY when there is no parent
    remote_parent_sampled=ALWAYS_ON,     # defaults — leave them alone
    remote_parent_not_sampled=ALWAYS_OFF,
    local_parent_sampled=ALWAYS_ON,
    local_parent_not_sampled=ALWAYS_OFF,
)

There is one important exception to “downstream rates do nothing”: a service that starts traces of its own. A consumer reading from a queue, a cron job, or an endpoint reachable without going through the gateway all create roots, and for those the local ratio is fully in effect. Most services are a mix, which is why measured volume rarely matches a naive prediction from the configured rate — see propagating trace context through Kafka consumers for the queue case.

Where the decision is made, and where it is only inherited Top row: the gateway decides at ten percent, then the checkout and payments services inherit that decision regardless of their own configured rates. Bottom row: a queue consumer has no incoming parent, so it creates a root span and its own rate of fifty percent applies. A rate only bites where a trace begins gateway · root rate 10% — applies checkout rate 1% — ignored payments always_on — ignored flag flag Kafka message no traceparent consumer · root rate 50% — applies Services that both serve requests and consume queues apply their rate to some traffic and not to the rest.

Writing a custom sampler, and when it is justified

The standard samplers cover volume control and nothing else. Two requirements regularly justify a custom one, and both are narrow enough to implement in a few dozen lines.

Rate by route rather than by service. A health-check endpoint hit twice a second by every probe in the cluster does not need any coverage; a payments endpoint needs a great deal. A custom root sampler that reads the span’s attributes and picks a ratio per route gives you that without deploying a separate service per rate.

from opentelemetry.sdk.trace.sampling import (
    Sampler, SamplingResult, Decision, TraceIdRatioBased, ParentBased)

class PerRouteSampler(Sampler):
    """Ratio chosen from http.route; falls back to a fleet default."""
    RATES = {"/healthz": 0.0, "/metrics": 0.0, "/checkout": 0.25}
    DEFAULT = 0.02

    def should_sample(self, parent_context, trace_id, name, kind=None,
                      attributes=None, links=None, trace_state=None):
        route = (attributes or {}).get("http.route", "")
        rate = self.RATES.get(route, self.DEFAULT)
        # Delegate to the standard ratio sampler so the trace-id derivation
        # stays identical to every other service in the fleet.
        return TraceIdRatioBased(rate).should_sample(
            parent_context, trace_id, name, kind, attributes, links, trace_state)

    def get_description(self):
        return "PerRouteSampler"

# Wrap it so downstream services still inherit the decision.
sampler = ParentBased(root=PerRouteSampler())

The attribute caveat is important: should_sample only sees attributes passed at span creation. An attribute set later — including http.route in frameworks that resolve the route after the span starts — is invisible to the sampler. Pass anything the sampler needs as a creation-time attribute, or the rule silently falls through to the default.

A debug override. A header or authenticated flag that forces sampling for a specific session lets support capture a full trace on demand. Guard it: an unauthenticated override is an amplification vector, since anyone can force 100% sampling of your fleet by setting a header on every request. Rate-limit it, require an internal token, and keep the override out of the path for anonymous traffic — the same trust reasoning as browser-supplied context.

Verification

Prove which service is deciding, rather than reasoning about configuration files:

# 1. Send an unsampled parent and confirm the downstream service drops it.
curl -H "traceparent: 00-$(openssl rand -hex 16)-$(openssl rand -hex 8)-00" \
     https://api.example.com/checkout/123
# Expect: nothing stored, whatever the downstream rate says.

# 2. Send a sampled parent and confirm it is kept.
TID=$(openssl rand -hex 16)
curl -H "traceparent: 00-${TID}-$(openssl rand -hex 8)-01" \
     https://api.example.com/checkout/123
curl -s "http://tempo:3200/api/traces/${TID}" | jq '.batches | length'
# Expect: a complete trace, even at a 1% configured rate.

If step 1 stores a trace anyway, some service is running a non-parent-based sampler and is making independent decisions — the usual cause of traces with missing middles.

Decision criteria

  • Setting a fleet rate? Change it at the services that start traces: the gateway, the ingress, the queue consumers. Downstream values are documentation, not control.
  • Want every trace from a specific low-traffic service? always_on works only if that service creates the root. Otherwise you need the caller to sample more, or tail-based sampling at the Collector.
  • Need error traces guaranteed? No head-based configuration can promise that; the decision precedes the error. Use tail sampling.
  • Running an untrusted client? Do not let a caller’s sampled flag decide your storage. Override remote_parent_sampled with a ratio at your trust boundary, and accept the resulting partial traces as the cost of the protection.

Common pitfalls

  • Assuming parentbased_always_on means “no sampling”. It means “keep every trace this service starts”. On an edge service that is 100% of traffic and a substantial bill.
  • Mixing traceidratio and parentbased_traceidratio in one fleet. The independent sampler will disagree with the parent whenever the rates differ, producing partial traces that look like propagation bugs.
  • Changing rates during an incident. New rates apply to new traces only, and the traces you want are the ones that already happened. Raise the rate to catch the next occurrence, not this one.
  • Forgetting internal callers. A health check, a cron sweep, or an internal admin tool that bypasses the gateway starts its own traces and is governed by that service’s own rate.

Related

↑ Back to Choosing Between Head-Based and Tail-Based Sampling