Debugging Invalid traceparent Headers

A traceparent must be exactly 00- plus 32 lowercase hex characters, a hyphen, 16 lowercase hex characters, a hyphen, and two hex characters — with neither ID all zeros; anything else is discarded silently by the SDK and the request becomes a new trace root.

Context and when it matters

The word silently is what makes this worth a page. A malformed traceparent does not raise, does not log at default levels, and does not mark the span. The SDK simply ignores the header and starts a fresh trace, producing the fragmentation pattern where the caller’s trace ends at a hop and a brand-new trace begins on the other side with a different ID.

Malformed headers come from a small set of sources: hand-rolled propagation code that builds the string by concatenation, uppercase hex from a language whose formatter defaults that way, a gateway that truncates long header values, a test harness sending a placeholder, and — most often — a legacy bridge converting some other format without padding IDs to the right width, as covered in the B3 migration guide.

The validation rules, field by field

The four fields of a traceparent and their constraints The header value is split into four labelled segments. Version is two hex characters and must currently be 00. Trace ID is thirty-two lowercase hex characters and must not be all zeros. Parent span ID is sixteen lowercase hex characters and must not be all zeros. Flags is two hex characters where bit zero is the sampled flag. Total length is always fifty-five characters. Always 55 characters · always lowercase 00 4bf92f3577b34da6a3ce929d0e0e4736 00f067aa0ba902b7 01 version 2 hex must be 00 trace id 32 hex, lowercase not all zeros parent span id 16 hex, lowercase not all zeros flags 2 hex bit 0 = sampled Version 00 is strict; future versions are forwards-compatible A parser seeing a higher version must accept the first four fields and ignore any trailing ones, rather than reject the header. A parser seeing version 00 must reject anything with extra fields — so "00-…-01-extra" is invalid, while "01-…-01-extra" is not. Version ff is reserved and always invalid.

Implementation

Capture and classify at the boundary

import re

TRACEPARENT_RE = re.compile(r"^00-([0-9a-f]{32})-([0-9a-f]{16})-[0-9a-f]{2}$")
ZERO_TRACE, ZERO_SPAN = "0" * 32, "0" * 16

def classify_traceparent(raw: str | None) -> str:
    """Return a bounded reason code — the value you record and alert on."""
    if raw is None:
        return "absent"
    if len(raw) != 55:
        # Truncation by a proxy is the usual cause; so is an extra field.
        return "bad_length"
    m = TRACEPARENT_RE.match(raw)
    if not m:
        if raw[:3] != "00-":
            return "bad_version"
        if raw != raw.lower():
            return "uppercase_hex"       # valid hex, invalid per spec
        return "bad_charset"             # non-hex characters present
    trace_id, span_id = m.groups()
    if trace_id == ZERO_TRACE:
        return "zero_trace_id"           # placeholder from a test harness
    if span_id == ZERO_SPAN:
        return "zero_span_id"            # common in hand-rolled bridges
    return "valid"

Wire it into middleware so the classification is recorded on every request rather than discovered during an incident:

@app.middleware("http")
async def traceparent_guard(request, call_next):
    reason = classify_traceparent(request.headers.get("traceparent"))
    span = trace.get_current_span()
    if reason not in ("valid", "absent"):
        # Bounded value: six or seven possible reasons, safe to aggregate.
        span.set_attribute("propagation.reject_reason", reason)
        logger.warning("invalid traceparent (%s) from %s on %s",
                       reason, request.client.host if request.client else "?",
                       request.url.path)
    return await call_next(request)

The six failure shapes and what each means

Six ways a traceparent goes wrong, and where each comes from Six rows. Wrong length usually comes from proxy truncation or a 64-bit trace ID and is fixed by padding at the bridge. Uppercase hex comes from a formatter default and is fixed by lowercasing at the source. An all-zero trace ID comes from a test harness or placeholder and should be rejected. An all-zero span ID comes from a hand-rolled bridge that had no parent to reference. A wrong version comes from a hard-coded string. Non-hex characters come from string concatenation with an unset variable. Each shape points at a different source reason usual source fix bad_length 64-bit id, or proxy truncation pad at the bridge uppercase_hex formatter default (Java, Go) lowercase at source zero_trace_id test harness placeholder reject, do not adopt zero_span_id bridge with no real parent generate a root span id bad_version hard-coded literal, ff reserved use the SDK injector bad_charset concatenation with unset var stop building it by hand Five of the six vanish permanently once the sender uses its SDK's injector instead of building the string.

Reproduce it from the command line

# A quick harness for each shape — run against a staging endpoint and watch
# which ones produce a fresh trace id downstream.
BASE="https://staging.example.com/api/health"
declare -A CASES=(
  [valid]="00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"
  [uppercase]="00-4BF92F3577B34DA6A3CE929D0E0E4736-00F067AA0BA902B7-01"
  [short_id]="00-4bf92f3577b34da6-00f067aa0ba902b7-01"
  [zero_trace]="00-00000000000000000000000000000000-00f067aa0ba902b7-01"
  [zero_span]="00-4bf92f3577b34da6a3ce929d0e0e4736-0000000000000000-01"
  [extra_field]="00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01-extra"
)
for name in "${!CASES[@]}"; do
  code=$(curl -s -o /dev/null -w '%{http_code}' -H "traceparent: ${CASES[$name]}" "$BASE")
  echo "${name}: HTTP ${code} — check whether the server span kept the trace id"
done

What the SDK does with each shape, and why it stays quiet

Understanding the SDK’s reaction explains why these defects survive for months. The W3C specification is explicit that a receiver encountering an invalid traceparent must behave as though the header were absent — it must not error, must not partially adopt the value, and must not propagate it onward. Every compliant SDK therefore discards the header and starts a new trace, which is exactly the correct behaviour and exactly what makes the problem invisible.

The consequence is that “trace starts here” and “trace was broken here” look identical in storage. Both produce a root span with no parent. The only way to distinguish them is at the boundary, before the SDK’s parser has thrown the evidence away — which is why the middleware above records a reason code rather than relying on anything the trace itself can tell you later.

There is one shape that behaves differently and deserves separate attention: a header that is valid but semantically wrong. A traceparent with a well-formed but reused trace ID — a client that generates one ID at startup and sends it forever, or a load test that hard-codes a value — passes every validation rule. The SDK adopts it happily, and the result is a single trace accumulating millions of spans until no UI can render it. Nothing is malformed; the data is simply unusable. Detecting this needs a different check: alert on the span count of the largest trace in a window, or on any single trace ID appearing at a rate incompatible with real traffic.

The two checks together — reason codes for malformed headers, span-count ceilings for valid-but-abused ones — cover essentially every way inbound context goes wrong. Both are cheap, both are permanent, and both turn an invisible defect into a number someone can watch.

Three outcomes for an inbound traceparent Three rows. A valid header produces a span that continues the caller's trace. A malformed header is discarded silently and produces a new root span, indistinguishable in storage from a genuine trace start. A valid but reused header is adopted and appends to an ever-growing trace that eventually cannot be rendered. Only one of these three is visible without a boundary check valid header → span joins the caller's trace correct malformed → discarded, new root created silent valid but reused → adopted, trace grows without bound worse Reason codes catch row two; a span-count ceiling catches row three.

Verification

  • propagation.reject_reason is absent from production traffic. Any non-zero rate is a real sender to fix, not noise.
  • A known trace ID survives the hop. Send the valid case above and confirm the downstream span carries the same trace ID.
  • The malformed cases start new traces rather than erroring. That is correct SDK behaviour; the goal is visibility, not rejection of the request.
  • No 4xx is returned for a bad header. Never fail a user request over telemetry — instrumentation must degrade, not break the product.

Where to fix it, and in what order

A malformed header always has two possible fix sites, and picking the wrong one first is why these defects recur.

The source is the correct fix. A sender building the header by string concatenation, formatting hex in uppercase, or padding incorrectly will keep doing so for every downstream service it talks to, not just the one that reported the problem. Replacing hand-rolled injection with the SDK’s propagator removes the entire class of defect in one change, and it is almost always a smaller diff than the hand-rolled version it deletes.

The boundary is the correct mitigation while the source is being fixed. Normalising at a gateway — lowercasing hex, padding a short trace ID, dropping an unusable value — restores continuity immediately for the traffic passing through that gateway. It does not help any path that bypasses it, which is why it is a mitigation rather than a fix.

Do both, in that order of priority but usually in parallel: mitigate at the boundary today so traces reconnect, then fix the sender this sprint so the mitigation can eventually be removed. What you should avoid is the third option, which teams reach for surprisingly often — making the receiver more permissive. Accepting uppercase hex or short IDs in your own extractor works until a request reaches a service that has not been patched, and it leaves the fleet with two different definitions of a valid header. The specification’s strictness is doing you a favour: it guarantees that every compliant service agrees on what a valid context is.

One practical exception: if the non-compliant sender is a third party you cannot change — a partner integration, a vendor’s webhook, a device fleet you do not control — then boundary normalisation is the permanent fix. Put it at the ingress, document it, and record a counter so you can tell when the third party finally fixes their side.

Common pitfalls

  • Rejecting the request instead of the header. Turning a malformed traceparent into a 400 converts a telemetry defect into an outage. Drop the header, serve the request.
  • Adopting a zero trace ID. It is explicitly invalid, and adopting it merges unrelated requests into one enormous unusable trace.
  • Fixing it only at the boundary. Normalising in a gateway restores the trace but leaves the broken sender in place, and the next endpoint it calls will break too.
  • Logging the raw header at high volume. Under a broken deployment this floods logs. Log the bounded reason code and sample the raw value.
  • Treating uppercase as harmless. It is valid hexadecimal and invalid per specification; SDKs reject it, so it breaks traces exactly like a malformed value.

Related

↑ Back to Understanding W3C TraceContext Propagation