Diagnosing Missing and Broken Traces

Problem Framing

An engineer pastes a trace ID into the UI and gets “trace not found”. Or worse: the trace is there, but it stops at the third service, and the two services downstream — which definitely ran, because the order was placed — contributed nothing. Either way the investigation stalls, and the usual response is to guess: restart something, bump a queue size, add a log line.

Missing telemetry has a small number of causes and a fixed pipeline in which to look for them. A span has to be created, sampled, exported, accepted, forwarded, and indexed, and every one of those steps has a counter that says whether it happened. This page walks the pipeline in order, because checking it in order is what turns a two-hour guessing session into a five-minute diagnosis.

Prerequisites

  • Access to the service’s logs and environment, ideally with the ability to set OTEL_LOG_LEVEL=debug on one instance.
  • Access to the Collector’s own metrics endpoint (:8888/metrics by default).
  • Query access to the trace backend, including fetch-by-ID — see querying traces with TraceQL and Jaeger search.
  • Knowledge of the sampling configuration, because a “missing” trace is frequently a correctly sampled-out trace.

Concept Deep-Dive: Six Places a Span Can Die

The span pipeline and the counter that proves each stage Six stages left to right: created, sampled, queued, exported, collector, indexed. Under each stage is the characteristic failure and the metric that reveals it: instrumentation not applied, sampler dropped, queue full, exporter failure, refused or dropped by processor, and index or clock-skew problems. Walk it in order — each stage has a counter that says yes or no 1 Created SDK + instr. 2 Sampled sampler decision 3 Queued batch processor 4 Exported OTLP over wire 5 Collector accept + forward 6 Indexed backend storage init order wrong library not patched rate too low parent not sampled queue full no shutdown flush TLS / DNS failure wrong endpoint memory_limiter filter dropped it clock skew retention expired The counter that proves the stage console exporter sampler config queue_size export failures refused_spans fetch by trace id Fragmented traces (each service its own root) are a different failure class: nothing died, context simply did not travel. Diagnose those at the boundary — inbound headers on the callee — not in the export pipeline.

Before touching anything, classify what you are looking at. The three shapes have almost disjoint cause sets:

  • Absent — nothing from a service, ever. Instrumentation or export configuration.
  • Partial — traces exist but some spans are missing, often the deepest or last ones. Queue pressure or sampling.
  • Fragmented — every service produces traces, but each is its own root. Context propagation, not export.

Step-by-Step Implementation

Step 1 — Prove the span exists in the process

Take transport out of the picture entirely:

# Restart one instance with the console exporter and issue exactly one request.
OTEL_TRACES_EXPORTER=console \
OTEL_LOG_LEVEL=debug \
OTEL_SERVICE_NAME=checkout-api \
  python -m myservice

If spans print, creation works and the problem is downstream. If nothing prints, the cause is almost always one of three things: the SDK initialized after the framework was imported (so nothing was patched), the sampler is always_off or a rate rounding to zero, or the instrumentation package for that library is not installed. OTEL_LOG_LEVEL=debug will name the instrumentations it actually loaded.

Step 2 — Rule out sampling

A trace that was never sampled is not a bug, and confirming this takes seconds:

# What did this process decide?
env | grep -E 'OTEL_TRACES_SAMPLER|OTEL_TRACES_SAMPLER_ARG'
# OTEL_TRACES_SAMPLER=parentbased_traceidratio
# OTEL_TRACES_SAMPLER_ARG=0.01        ← 1 in 100 traces is stored

With parentbased_* the upstream decision wins, so a service that looks “unsampled” may simply be honouring a caller that sampled out. Force one trace through instead of arguing about probabilities:

# Send a request with the sampled flag set (last byte 01) and a known trace id
curl -H "traceparent: 00-11111111111111111111111111111111-2222222222222222-01" \
     https://api.example.com/checkout/123
# Then fetch that exact id from the backend — no search, no time window.
curl -s http://tempo:3200/api/traces/11111111111111111111111111111111 | jq '.batches | length'

Step 3 — Check the SDK’s export path

# Debug logging names the endpoint and reports failures explicitly
OTEL_LOG_LEVEL=debug ./service 2>&1 | grep -iE 'otlp|export|refused|deadline'
# Failed to export spans: connection refused: otel-collector:4317

The classic misconfigurations, in the order they occur in practice: exporting gRPC (4317) to an HTTP-only receiver (4318) or the reverse; a https:// endpoint against a plaintext receiver; a OTEL_EXPORTER_OTLP_ENDPOINT that includes /v1/traces when the SDK appends it too; and a NetworkPolicy that blocks the collector port while allowing everything else.

Step 4 — Read the Collector’s own counters

The Collector is the only component in the chain that publishes a complete accounting of what it received, refused, dropped, and sent.

curl -s http://otel-collector:8888/metrics | grep -E \
  'receiver_accepted_spans|receiver_refused_spans|processor_dropped_spans|exporter_sent_spans|exporter_send_failed_spans|exporter_queue_size'

# otelcol_receiver_accepted_spans{receiver="otlp"}        1284391
# otelcol_receiver_refused_spans{receiver="otlp"}            9822   ← memory_limiter
# otelcol_processor_dropped_spans{processor="filter"}       41200   ← filtered on purpose?
# otelcol_exporter_send_failed_spans{exporter="otlp/tempo"}   118
# otelcol_exporter_queue_size{exporter="otlp/tempo"}          4998  ← near capacity

Read those numbers as a funnel. The gap between what a service emitted and what the backend stored is never a mystery once each drop is attributed:

Span funnel for one hour, with each loss attributed Four descending bars. Emitted by services: 1,420,000 spans. Accepted by the collector: 1,284,391, losing 135,609 to memory limiter refusals. After processors: 1,243,191, losing 41,200 to a filter processor. Stored in the backend: 1,243,073, losing 118 to export failures. One hour · every missing span accounted for emitted by SDKs 1,420,000 accepted by collector 1,284,391 after processors 1,243,191 stored in backend 1,243,073 −135,609 memory limiter · −41,200 filter processor · −118 export failures A drop with no matching counter is the one worth investigating — it means the span died before the collector saw it.

refused_spans climbing means the memory limiter is shedding load — the fix is capacity or sampling, not a config tweak on the SDK. processor_dropped_spans on a filter processor usually means a filter matched more than its author intended.

Step 5 — Distinguish “not stored” from “not found”

# Fetch by ID bypasses search entirely.
curl -s -o /dev/null -w '%{http_code}\n' \
  http://tempo:3200/api/traces/4bf92f3577b34da6a3ce929d0e0e4736
# 200 → stored, so the problem is the search query or the time window
# 404 → never stored, go back to step 4

If fetch-by-ID works but search does not, the cause is one of: a time range that excludes the trace because of clock skew, a service.name that is missing (so the trace is filed under unknown_service), or an attribute filter naming a key nothing sets.

Which spans go missing first, and why that is a clue

Partial traces are not random. The pattern of what survives tells you which stage failed, and reading it correctly saves a step:

  • The deepest spans are missing. Queue pressure. The batch processor drops on insert when the queue is full, and the deepest spans are the last created in a request, so they arrive at a queue that is already saturated. Raise max_queue_size, shorten schedule_delay_millis, or reduce span volume per request.
  • One service is missing everywhere. Configuration, not load. That service exports to a different endpoint, runs a different sampler, or never had instrumentation applied. Compare its environment against a working peer rather than tuning anything.
  • Spans are missing only at high traffic. Capacity. Correlate the gaps with otelcol_receiver_refused_spans and with the service’s own CPU; a memory limiter shedding load looks exactly like intermittent instrumentation bugs from the UI.
  • Only long-running requests are truncated. Export timeouts. A request that outlives the exporter’s timeout gets its later spans dropped when the connection is torn down, particularly with streaming or websocket handlers.
  • Only the last request before a deploy is incomplete. Shutdown flush. The pod terminated before the processor drained; add a preStop hook with a few seconds of grace, or call shutdown() on SIGTERM.

A useful discipline: before changing any configuration, write down which of these five patterns you are seeing. If the observation does not match any of them, the cause is more likely fragmentation (context loss) than loss, and the next section applies instead.

Diagnosing a fragmented trace

Fragmentation is a boundary problem, so inspect the boundary. Log the inbound header on the callee before anything else touches it:

# Temporary middleware — prove what actually arrived
@app.middleware("http")
async def log_inbound_context(request, call_next):
    tp = request.headers.get("traceparent")
    logger.info("inbound traceparent=%s tracestate=%s",
                tp or "<ABSENT>", request.headers.get("tracestate", "-"))
    return await call_next(request)

Four outcomes, four causes:

Reading the inbound traceparent to locate the break Four rows. Row one: header absent — the caller never injected it or an intermediary stripped it. Row two: header present but malformed — the callee rejects it and starts a new root. Row three: header valid but its trace ID differs from the caller's — context was lost inside the caller before the request went out. Row four: header correct — the break is downstream of this hop. What the callee sees → where the break is observed inbound header cause fix lives in <ABSENT> never injected, or stripped by a proxy caller / proxy 00-4bf9…-00000…-01 malformed — all-zero id, bad length caller SDK valid, unexpected id context lost in caller before send async boundary valid, expected id propagation is fine — export is not callee export One log line at the boundary replaces an afternoon of guessing which side of the hop is at fault.

Verification

You have found the cause, not a symptom, when you can state the stage and show the counter that proves it:

  • “Spans die at stage 3 — queue_size sits at max_queue_size and the SDK’s dropped counter climbs at 400/s.”
  • “Spans die at stage 5 — receiver_refused_spans tracks memory limiter activations one-for-one.”
  • “Nothing died — the callee logs <ABSENT> for traceparent, so the gateway is stripping it.”

Then re-run the exact reproduction from Step 2 and confirm the trace is complete end to end.

Making the next occurrence a five-minute job

Most of the time spent on a missing-trace investigation goes into establishing facts that a small amount of standing instrumentation would have already answered. Three cheap additions pay for themselves the first time they are used.

Publish the pipeline counters on a dashboard nobody has to build during an incident. Accepted, refused, dropped, sent, and failed spans per collector, plus SDK queue depth where the SDK exposes it. The shape of those five lines identifies the failing stage before anyone opens a terminal.

Emit a synthetic trace on a schedule. A tiny job that issues one request per minute through the full request path with a known, forced-sampled context, then queries the backend for it thirty seconds later, converts “traces are missing” from a report into an alert. It also catches the silent failures — an expired TLS certificate on the collector, a NetworkPolicy change, a retention setting adjusted by someone else — that only ever surface when a human happens to look.

Log the inbound traceparent at the edge, permanently, at a low sample rate. One structured log line per sampled request recording whether the header was present, well-formed, and which trace ID it carried costs almost nothing and answers the single most common question in these investigations without a redeploy.

The common thread is that every check in this page is a fact you can capture continuously rather than reconstruct under pressure. A team that has these three in place typically spends the investigation confirming which stage lost the span, not arguing about whether spans are being lost at all.

Edge Cases and Gotchas

  1. Short-lived processes lose everything. A CLI, a Lambda, or a job that exits without calling shutdown() discards whatever is queued. Register a shutdown hook and, for serverless, use a simple exporter or an explicit flush before returning.
  2. Two SDKs in one process fight. A Java agent plus a manually configured SDK produces two providers, and only one wins GlobalOpenTelemetry. Spans go to whichever registered first — often the one you did not configure.
  3. Sampling looks like loss during an incident. Under load-shedding, some setups reduce sample rates automatically. Traces genuinely vanish, correctly, exactly when you want them most.
  4. The trace exists but is far too big to render. A retry loop that reuses one trace ID can produce a trace with 200,000 spans; the UI times out and reports nothing. Query the span count before assuming absence.
  5. Retention silently deletes yesterday’s evidence. A 24-hour retention means the incident from last night is already gone. Confirm the retention window before searching for old traces.
  6. Clock skew hides spans in plain sight. A node ten minutes fast writes spans stamped in the future; a search over “the last five minutes” misses them, though fetch-by-ID finds them immediately.

Performance and Scale Notes

Queue sizing is the most common real fix. max_queue_size defaults to 2048 spans. A service producing 60 spans per request at 300 RPS generates 18,000 spans/second, so the queue turns over every 114 ms — any exporter hiccup drops spans. Size it to at least peak_spans_per_second × schedule_delay_seconds × 1.5.

Debug logging is not free. OTEL_LOG_LEVEL=debug on a busy service can cost more CPU than the tracing itself. Enable it on one instance behind a load balancer, not fleet-wide.

The console exporter is a diagnostic tool, not a fallback. It serializes every span to stdout synchronously and will change the latency profile of the service you are debugging.

Collector metrics are your early-warning system. Alert on otelcol_receiver_refused_spans and otelcol_exporter_send_failed_spans being non-zero; both are silent by default, which is why missing traces are usually discovered by a human days later.

Troubleshooting FAQ

My service produces no traces at all — where do I start?

Console exporter, one request. Spans printing means the problem is transport or backend; nothing printing means instrumentation never ran, usually an initialization-order or sampler problem.

Why do I see the caller’s span but not the callee’s?

Either the header never arrived or the callee exports somewhere else. Log the inbound traceparent at the callee before anything else — that one line splits the problem in half. Detail in debugging traces that never reach the backend.

Traces appear but each service is its own separate trace. Why?

Extraction is failing: mismatched propagators, a malformed header, or context lost across an async boundary in the caller. See fixing broken parent-child links across services.

Some traces are complete and others are truncated. What causes that?

Queue pressure. The batch processor drops what arrives when the queue is full, so long traces lose their tails. Check the dropped-span counter first.

The trace exists in the backend but the UI shows nothing. Why?

Clock skew putting spans outside the searched window, or a missing service.name. Fetch by trace ID to separate a storage problem from a search problem.

↑ Back to Trace Debugging & Signal Correlation