Debugging Traces That Never Reach the Backend

Switch one instance to the console exporter and issue a request: if spans print, the SDK is fine and the problem is endpoint, protocol, TLS, or network policy — and if nothing prints, no amount of pipeline debugging will help because the spans were never created.

Context and when it matters

“No traces from this service” has two completely different root causes that look identical from the trace UI, and the console exporter separates them in under a minute. Skipping that step is the most common reason these investigations take hours: people begin by checking the Collector, find it healthy, check the network, find it fine, and eventually discover the SDK never instrumented anything.

Once creation is confirmed, the export path has a small number of failure points, and each has a decisive test. This page walks them in the order that eliminates the most possibilities per step.

The export path

Each hop on the export path and how to test it Six stages left to right: span creation tested by the console exporter, the batch queue tested by SDK debug logs, the exporter tested by its error output, the network tested with a direct OTLP request from inside the pod, the collector receiver tested by its accepted spans counter, and the backend tested by fetching a known trace ID. Test each hop; stop at the first failure created queued exported network received stored console exporter prints or not OTEL_LOG_LEVEL =debug exporter errors in stderr curl from the pod to :4318 accepted_spans rising? fetch by id 200 or 404 The first three tests run inside the process and need no infrastructure access. The last three need the Collector's metrics endpoint and the backend's API — worth having to hand before an incident. Working left to right eliminates roughly half the remaining possibilities at each step.

The tests, in order

1. Do the spans exist?

OTEL_TRACES_EXPORTER=console OTEL_LOG_LEVEL=debug ./service
# Issue one request, then look for span JSON on stdout.

Nothing printed means creation failed, and the causes are: the SDK initialized after the framework was imported, the sampler is effectively zero, the instrumentation package is missing, or — for forking servers — the provider lives in the master process rather than the worker. That last one is covered for Django in instrumenting Django with OpenTelemetry and applies equally to gunicorn, uWSGI, and Celery.

2. Do endpoint, protocol, and port agree?

env | grep OTEL_EXPORTER
# OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
# OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf     ← mismatch: 4317 is gRPC

The three recurring mismatches: gRPC protocol against the HTTP port (4318) or the reverse (4317); an endpoint including /v1/traces when the SDK appends it, producing /v1/traces/v1/traces; and https:// against a plaintext receiver, which fails during TLS handshake with an error that mentions neither TLS nor tracing.

3. Can the pod reach the Collector?

kubectl exec -it deploy/checkout-api -- sh -c '
  # An empty OTLP payload is enough to prove reachability — expect 200 or 400,
  # both of which mean the receiver answered. Connection refused means it did not.
  curl -sS -o /dev/null -w "%{http_code}\n" \
    -X POST http://otel-collector:4318/v1/traces \
    -H "Content-Type: application/json" -d "{}"'

A hang rather than a refusal usually means a NetworkPolicy is dropping the packets silently, which is a different fix from a wrong hostname and is worth distinguishing before anyone edits the deployment.

4. Is the Collector accepting them?

curl -s http://otel-collector:8888/metrics \
  | grep -E 'receiver_accepted_spans|receiver_refused_spans|exporter_send_failed'

Accepted rising while the backend stays empty points downstream; refused rising points at the memory limiter and is a capacity problem, covered in handling collector backpressure.

5. Did the process exit before flushing?

Why short-lived processes lose everything Two timelines for a job lasting three seconds. Without a shutdown flush, spans accumulate in the batch queue whose scheduled delay is five seconds, and the process exits before any export happens, discarding all of them. With a shutdown flush, the exit triggers an export and the spans arrive. Job runs 3 s · batch delay is 5 s no flush job runs · spans queue up exit — queue discarded first scheduled export would have been here with flush job runs · spans queue up shutdown() exports, then exits Affects CLIs, cron jobs, serverless functions, migrations, and any pod terminated without a grace period. In Kubernetes, add a preStop delay long enough for the flush to complete. 
import atexit, signal
from opentelemetry import trace

provider = trace.get_tracer_provider()

def _flush(*_):
    # Blocks until the queue drains or the timeout expires.
    provider.shutdown()

atexit.register(_flush)
signal.signal(signal.SIGTERM, lambda *_: (_flush(), exit(0)))

Verification

Once fixed, prove it end to end with a forced trace rather than by waiting for one to appear:

TID=$(openssl rand -hex 16)
curl -H "traceparent: 00-${TID}-$(openssl rand -hex 8)-01" \
     https://api.example.com/checkout/123
sleep 10
curl -s -o /dev/null -w '%{http_code}\n' "http://tempo:3200/api/traces/${TID}"
# 200 — the whole path works, end to end

Keep that as a synthetic check on a schedule. It converts the entire class of problem from “someone notices weeks later” into an alert.

The failures that only appear under load

Three export problems are invisible in a quiet environment and reliable in a busy one, which is why “it works in staging” is such a common part of this investigation.

Queue overflow. The batch processor’s queue is sized in spans, and at low traffic it never fills. At production rates it turns over in a fraction of a second, so any exporter hiccup — a brief network blip, a Collector restart, a slow backend — overflows it and drops whatever arrives next. The symptom is intermittent partial traces rather than complete absence, and the diagnostic is the SDK’s dropped-span counter rather than any error message.

Export timeouts under concurrency. An exporter with a default timeout and a single consumer serialises every batch. If the backend’s latency rises, batches queue behind each other and the effective export rate collapses well before the backend is actually saturated. Raising the consumer count is usually a bigger win than raising the queue size, and the two are frequently confused.

Connection churn. Some deployments recreate the exporter connection per batch — a misconfigured client, an aggressive idle timeout on a load balancer between the SDK and the Collector, or a service mesh terminating idle connections. Each batch then pays TCP and TLS setup, which at high batch rates is a substantial and entirely invisible cost. The tell is export latency that is high and constant regardless of payload size.

All three share a diagnostic property: they show up in counters and not in logs. Exporting the SDK’s own metrics — queue size, dropped spans, export duration — is the single highest-value addition you can make to a pipeline that is intermittently losing data, and it takes one processor and a scrape endpoint.

When the answer is “it is working correctly”

A meaningful share of these investigations end with the pipeline being fine and the expectation being wrong. Three cases account for most of them.

The trace was sampled out, which at a 1% rate is the overwhelmingly likely outcome for any single request. The trace exists but under a different service name, because service.name was unset and the backend filed it under unknown_service. Or the search window excluded it, because a clock offset put the spans minutes away from where the query looked.

All three are worth checking before escalating, and all three are ruled out by the forced-trace test: a request with an explicitly sampled traceparent, fetched by ID rather than searched. If that works, the pipeline is healthy and the question becomes which of the three expectations was wrong — a much shorter conversation than a capacity investigation.

Endpoint and protocol combinations that fail gRPC exporter → :4318 *fails 4318 is HTTP HTTP exporter → :4317 *fails 4317 is gRPC https:// → plaintext receiver fails at handshake error mentions TLS, not tracing endpoint with /v1/traces *double path SDK appends it too correct pairing works gRPC:4317, HTTP:4318 Endpoint and protocol combinations that fail configuration result note gRPC exporter → :4318 fails 4318 is HTTP HTTP exporter → :4317 fails 4317 is gRPC https:// → plaintext receiver fails at handshake error mentions TLS, not tracing endpoint with /v1/traces double path SDK appends it too correct pairing works gRPC:4317, HTTP:4318 Four of these five produce an error that never mentions the actual mistake.

Keep the reproduction, not the fix

Whatever the cause turns out to be, the artefact worth keeping is the forced-trace reproduction rather than the configuration change. A one-line script that sends a request with a known, sampled traceparent and then fetches that ID from the backend is the fastest possible answer to “is the pipeline healthy right now”, and it works identically whether the last failure was a wrong port, a NetworkPolicy, or a missing shutdown flush.

Run it from inside the cluster on a schedule and alert when it fails. That single check covers every stage on this page — creation, sampling, export, transport, acceptance, storage — because a trace that completes the round trip proves all six worked. It is the cheapest observability-of-observability you can build, and it turns the entire class of problem from an investigation into a notification.

Common pitfalls

  • Debugging the Collector first. Half of these investigations end at the console exporter, which takes a minute and needs no infrastructure access.
  • Testing connectivity from the wrong place. A curl from your laptop proves nothing about a pod behind a NetworkPolicy; run it from inside the workload.
  • Ignoring the SDK’s own error output. The exporter logs the reason, at debug level, and the message usually names the exact problem.
  • Assuming a restart fixed it. Restarting clears the symptom while the queue refills; verify with a forced trace instead.
  • Forgetting sampling. A service configured at 1% produces one trace in a hundred, and the first few requests you test may legitimately produce nothing.

Related

↑ Back to Diagnosing Missing and Broken Traces