Fixing Broken Parent-Child Links Across Services

Log the inbound traceparent at the callee before anything else: absent means the caller or an intermediary did not send it, malformed means the caller built it wrongly, and a valid-but-unexpected trace ID means the caller lost context internally before making the call.

Context and when it matters

A fragmented trace is different from a missing one. Every service is producing spans, exports are healthy, and the Collector counters are clean — but instead of one trace spanning five services you have five traces of one service each. Nothing is broken in the export pipeline; the context is simply not travelling.

This is the failure that wastes the most time, because both teams’ evidence says their side works. The caller’s trace is complete up to the point it made the call; the callee’s trace is complete from the moment it received one. Neither can see the boundary, and the boundary is where the answer is.

One log line at the callee resolves it, and the rest of this page is what to do with each of the three answers it can give.

The three answers

What the inbound header tells you From a single log line at the callee, three branches. Header absent means the caller never injected or an intermediary stripped it, and the fix is in the caller or the proxy. Header malformed means the caller built the string by hand, and the fix is to use the SDK injector. Header valid but with an unexpected trace ID means the caller lost context internally, and the fix is at the caller's async or thread boundary. One log line, three diagnoses log at the callee inbound traceparent=? absent never sent, or stripped malformed rejected by the parser valid, wrong trace id caller lost its context fix: caller injection or proxy propagator config · header allow-list fix: use the SDK injector stop building the header by hand fix: caller's async boundary capture and restore context Without this line, all three look identical from either side: a complete trace that stops, and a new one that starts. 
# Temporary middleware at the callee. Remove it once the cause is found —
# or keep a sampled version permanently, which is cheaper than re-adding it.
@app.middleware("http")
async def log_inbound_context(request, call_next):
    tp = request.headers.get("traceparent")
    logger.info("inbound traceparent=%s b3=%s from=%s path=%s",
                tp or "<ABSENT>",
                request.headers.get("b3") or request.headers.get("x-b3-traceid") or "-",
                request.headers.get("user-agent", "-")[:40],
                request.url.path)
    return await call_next(request)

Logging the B3 headers alongside is deliberate: seeing <ABSENT> for traceparent and a populated x-b3-traceid immediately identifies a propagator mismatch, which is otherwise the hardest of the three to spot.

Cause 1: absent header

Two sub-causes, distinguished by whether the caller emitted it at all.

Propagator mismatch. The caller injects B3, the callee reads W3C, or vice versa. Common when a Spring Cloud Sleuth service talks to an OpenTelemetry one, or when an older Istio installation is in the path. The fix is a composite propagator on both sides, covered in the B3 to W3C migration guide:

export OTEL_PROPAGATORS=tracecontext,baggage,b3multi,b3

Stripped in transit. The caller sent it; something between removed it. Reverse proxies with header allow-lists, API gateways that rebuild requests, CDNs, and service meshes all do this by default in some configurations. Verify by capturing at both ends:

# At the caller's egress
kubectl exec -it deploy/checkout-api -- \
  sh -c 'tcpdump -A -s0 -i any port 8080 2>/dev/null | grep -i traceparent | head -3'

# At the callee's ingress — if it appears above and not here, an intermediary ate it.
kubectl exec -it deploy/inventory-api -- \
  sh -c 'tcpdump -A -s0 -i any port 8080 2>/dev/null | grep -i traceparent | head -3'

Cause 2: malformed header

The parser rejected it, so the callee behaves exactly as if it were absent. The validation rules and the six shapes this takes are covered in debugging invalid traceparent headers; the summary is that hand-built headers are the cause in nearly every case, and replacing the hand-rolled injection with the SDK’s propagator fixes the entire class.

Cause 3: valid header, unexpected trace ID

This one is the most interesting, because the network is working perfectly. The caller sent a well-formed header — for a different trace than the one it was serving. That means the context was lost inside the caller before the outbound request was made, and the outbound call started a new trace of its own.

Context lost inside the caller, before the call went out The caller's server span is active, then work moves to a thread pool or a detached promise where the context is not carried. The outbound HTTP client runs there with an empty context, generates a fresh trace ID, and sends a valid header for a trace nobody else knows about. The header is valid — it just belongs to the wrong trace server span trace 4bf92f… thread pool / promise context not captured HTTP client · new root trace 91ac33… the break happens here — not at the network The callee's log shows a valid header, so the callee is exonerated and the caller looks fine in isolation. The tell is that the trace ID in the callee's log does not appear in the caller's own trace at all. Fix at the boundary: capture the context at submission and restore it inside the task.

The fixes are per-language and covered in depth elsewhere: propagating context across thread pools in Java, and handling async boundaries in Node.js and Python for the runtime equivalents.

Verification

# Send a known trace id through the whole chain and confirm it survives.
TID=$(openssl rand -hex 16)
curl -H "traceparent: 00-${TID}-$(openssl rand -hex 8)-01" \
     https://api.example.com/checkout/123
sleep 10
curl -s "http://tempo:3200/api/traces/${TID}" \
  | jq -r '[.batches[].resource.attributes[]
            | select(.key=="service.name") | .value.stringValue] | unique'
# Expect every service in the path. A short list names the hop that broke.

That list is the whole verification: it tells you not just whether the fix worked but exactly how far the context now travels.

Preventing fragmentation from recurring

Every fix on this page addresses one hop. Keeping the whole fleet connected needs three standing controls, none of which is expensive.

A fleet-wide propagator standard, enforced in configuration rather than documentation. Set OTEL_PROPAGATORS in the base image or the deployment template so a new service inherits it, rather than relying on each team to configure it correctly. The composite list costs nothing on services that only ever see one format and saves the investigation entirely on the ones that see two.

A boundary attribute, recorded permanently at low volume. The propagation.format attribute described in the B3 migration guide doubles as a fragmentation detector: a service suddenly reporting none where it previously reported w3c is a break, visible as a metric rather than as a complaint.

A synthetic end-to-end check. One request per minute through the full path with a known trace ID, followed by a query for that ID, verifies that context still travels every hop. It catches the class of change that nobody thinks of as a tracing change — a new gateway rule, a mesh upgrade, a proxy replacing another — which is precisely the class that causes fragmentation.

The reason these are worth standing up is that fragmentation is introduced by infrastructure changes far more often than by application changes, and infrastructure changes are made by people who have no reason to think about trace context. A check that fails within a minute puts the feedback where it belongs, while the change is still fresh.

Message queues break differently

Everything above assumes an HTTP hop. Across a broker the same three causes apply with one addition: the context has to be written into the message and read back out, and either side can be missing without the other noticing.

A producer that publishes without injecting produces messages that consumers cannot connect to anything, and the consumer looks broken. A consumer that never extracts ignores a context the producer did include, and the producer looks broken. The diagnostic is the same in spirit — inspect the message headers at the consumer before extraction — but the tooling is different, and message headers are easier to inspect than HTTP ones because they are usually visible in the broker’s own tooling. See propagating trace context through Kafka consumers for the injection and extraction details.

Fragmentation causes across 40 investigated hops Fragmentation causes across 40 investigated hops. propagator mismatch 18 hops; header stripped by a proxy 12 hops; context lost in the caller 9 hops; malformed header 1 hop Fragmentation causes across 40 investigated hops propagator mismatch 18 hops header stripped by a proxy 12 hops context lost in the caller 9 hops malformed header 1 hop Three quarters of breaks are configuration rather than code — which is why the boundary log finds them so fast.

Common pitfalls

  • Fixing the callee. Making the receiver more permissive — accepting uppercase hex, accepting B3 when the fleet standard is W3C — masks the problem locally and leaves the sender broken for everyone else.
  • Assuming the mesh forwards headers. Istio, Envoy, and most gateways have their own propagation configuration, and it is not implied by the application’s.
  • Blaming the network for cause 3. A valid header with an unexpected trace ID never involves the network at all.
  • Removing the boundary log after the fix. Keeping a sampled version costs almost nothing and makes the next occurrence a one-minute diagnosis.
  • Testing with a synthetic request that skips the proxy. If the break is in an intermediary, a direct call to the callee will not reproduce it.

Related

↑ Back to Diagnosing Missing and Broken Traces