Debugging orphaned spans in async workflows

Orphaned spans are child spans emitted without a resolvable parent in the trace graph — fix them by capturing the active context propagation context before every async yield and explicitly re-attaching it inside the spawned task.

Context and when it matters

Async runtimes — Python’s asyncio, Node.js’s event loop, Java’s virtual threads — decouple task scheduling from the initiating coroutine or thread. When a task is dispatched across a scheduler boundary the OpenTelemetry SDK’s implicit context storage (contextvars in Python, AsyncLocalStorage in Node.js) may not transfer automatically. The result is a child span whose parent_id field is either 0000000000000000 (zero) or references a span that was never exported, breaking the parent-child linkage that makes a trace meaningful.

This matters most in three patterns: fire-and-forget asyncio.create_task() calls, ThreadPoolExecutor submits inside async handlers, and message queue consumers (Kafka, RabbitMQ) that reconstruct a span from broker headers instead of inheriting one from a live coroutine. Understanding the full span lifecycle and parent-child relationships is the foundation; this page focuses on the async-specific failure modes and exact remediation steps.

How orphaned spans form: an annotated diagram

The diagram below shows the two execution paths — the broken path where context is not transferred and the fixed path where it is captured and re-attached explicitly.

Core mechanism: the minimal broken pattern

The break is easiest to see in a standalone asyncio example before layering in framework abstractions.

import asyncio
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

async def process_async_task():
    # No parent context was passed — the SDK sees no active span
    # in this coroutine's contextvars storage and starts a new root.
    with tracer.start_as_current_span("background_worker") as span:
        span.set_attribute("task.status", "completed")
        # span.parent is None — this span is orphaned

async def main():
    with tracer.start_as_current_span("request_handler"):
        # Fire-and-forget: context is NOT automatically inherited
        asyncio.create_task(process_async_task())
        await asyncio.sleep(0.1)

asyncio.create_task copies the current contextvars snapshot at the moment of creation, but framework-specific schedulers, thread-pool dispatches, and library-level wrappers frequently reset or discard that snapshot before the coroutine body runs.

Step-by-step diagnostic workflow

Step 1 — verify traceparent propagation across the boundary

Enable SDK debug logging and print a context snapshot on both sides of the yield point.

# Enable OpenTelemetry SDK debug logging (both Python and Node.js)
export OTEL_LOG_LEVEL=debug
export OTEL_TRACES_EXPORTER=logging

from opentelemetry import trace
from opentelemetry.context import get_current
import logging

def log_context_snapshot(label: str) -> None:
    ctx = get_current()
    span = trace.get_current_span(ctx)
    sc = span.get_span_context()
    logging.info(
        "[%s] trace_id=%032x span_id=%016x is_valid=%s",
        label, sc.trace_id, sc.span_id, sc.is_valid
    )

Run this immediately before create_task() and at the top of the async task body. If trace_id changes or is_valid becomes False, propagation is broken at that boundary.

Step 2 — isolate SDK context manager leaks

In Node.js, watch for warnings that indicate the AsyncLocalStorage store was not inherited:

export OTEL_LOG_LEVEL=debug
export OTEL_NODE_DEBUG=1

Look for log lines like Context detached unexpectedly or Active span context mismatch. These appear when the SDK’s async-hook patches are either missing or bypassed by a third-party scheduler (common in frameworks that implement their own task queues, such as some job runners built on pg-boss or bull).

Step 3 — query for orphans in your trace backend

Tempo and Jaeger both expose queries that surface spans whose recorded parent ID is the zero value:

# Grafana Tempo — find spans recorded with no parent
{ rootSpan = false && parentSpanID = "0000000000000000" }

Cross-reference the resulting trace_id values with application logs. If the parent span appears in logs but not in the trace backend, the issue is likely exporter buffering or sampling configuration, not context loss.

Implementation detail: explicit context passing

Explicit context passing eliminates all runtime-specific context copying behaviour. It adds roughly 0.5–2 µs per boundary, which is negligible against any meaningful async workload.

import asyncio
from opentelemetry import trace, context as otel_context

tracer = trace.get_tracer(__name__)

async def process_with_explicit_context(ctx: otel_context.Context) -> None:
    # Re-attach the captured context in the new coroutine's storage
    token = otel_context.attach(ctx)
    try:
        with tracer.start_as_current_span("background_worker_explicit") as span:
            # span.parent now resolves to request_handler — linkage is intact
            span.set_attribute("propagation.mode", "explicit")
    finally:
        # Always detach; skipping this leaks context into subsequent tasks
        otel_context.detach(token)

async def main() -> None:
    with tracer.start_as_current_span("request_handler"):
        # Snapshot the live context BEFORE yielding control
        active_ctx = otel_context.get_current()
        task = asyncio.create_task(
            process_with_explicit_context(active_ctx)
        )
        await task  # awaiting ensures the task completes; adjust for fire-and-forget

For thread pools, the handling async boundaries in Node.js and Python cluster covers the ThreadPoolExecutor wrapper pattern. For immediate patching without code changes, instrument the standard library’s threading module:

from opentelemetry.instrumentation.threading import ThreadingInstrumentor

# Call once at application startup, before any threads are spawned
ThreadingInstrumentor().instrument()

The ThreadingInstrumentor intercepts Thread.start() and ThreadPoolExecutor.submit() to snapshot and re-attach context automatically. Overhead is ~1–3% CPU per context switch; for pipelines above ~10 k RPS prefer explicit passing.

For context propagation across service meshes and message brokers, inject the traceparent header into every outbound message rather than relying on in-process context storage:

from opentelemetry import propagate

# When producing a Kafka message or publishing to a queue
headers: dict[str, str] = {}
propagate.inject(headers)           # populates "traceparent" and "tracestate"
producer.send("my-topic", value=payload, headers=list(headers.items()))

Decision criteria

• Use explicit context.attach()/detach() when: the async framework has its own task scheduler that does not guarantee contextvars inheritance; you are mixing asyncio with ThreadPoolExecutor; or you need zero-overhead propagation across high-throughput pipelines.
• Use ThreadingInstrumentor or framework-specific instrumentation when: your team owns no task-creation code and patching call sites is impractical; and your RPS is well below 10 k so the ~1–3% scheduler overhead is acceptable.
• Check sampler configuration first when: orphans appear only for a subset of traces. Tail-based sampling can drop parent spans while retaining children, producing false orphan alerts.

Common pitfalls

• Forgetting context.detach(token) in a finally block. If an exception propagates before detach, the stale context leaks into subsequent tasks reusing the same coroutine slot, corrupting their parent linkage.
• Relying on asyncio.create_task alone for propagation. The contextvars snapshot that create_task copies is taken from the calling coroutine’s current context, but framework wrappers around create_task sometimes clear or replace that snapshot before the task body runs — always verify with a context snapshot log.
• Using ParentBasedSampler without also setting a high root probability. If the root span is dropped by TraceIdRatioBased(0.01), the sampler drops all children too — but the orphan detector fires on child spans that were logged before the sampling decision propagated. Set root=TraceIdRatioBased(1.0) during debugging, then reduce once linkage is verified.

Troubleshooting FAQ

Why do spans become orphaned only in async code, not in synchronous handlers? Synchronous code runs in a single call stack, so the SDK’s thread-local or coroutine-local context storage is never interrupted. Async runtimes hand tasks off across scheduler boundaries, which can reset or fail to copy the active context unless you explicitly attach it inside the new execution unit.

Can tail-based sampling create false orphan alerts even when propagation is correct? Yes. If the sampler drops the parent span but retains the child, the child’s parent_id points to a non-existent span, triggering orphan alerts. Use ParentBasedSampler so children are only retained when their parent is also retained:

from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased

sampler = ParentBased(root=TraceIdRatioBased(0.1))

How do I detect orphaned spans in Tempo or Jaeger without inspecting every trace manually? In Grafana Tempo use { rootSpan = false && parentSpanID = "0000000000000000" }. In Jaeger, filter by parent_span_id=0 in the search UI or via the HTTP API’s tags parameter.

Does patching ThreadPoolExecutor with ThreadingInstrumentor add significant overhead? Roughly 1–3% CPU per context switch. For pipelines above ~10 k RPS, prefer explicit context.attach()/detach() pairs to avoid event-loop contention and GC pressure.

Validation test harness

Before shipping a fix, confirm it with a deterministic in-process test using the InMemorySpanExporter:

from opentelemetry.sdk.trace import TracerProvider, SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry import trace

exporter = InMemorySpanExporter()
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(exporter))
trace.set_tracer_provider(provider)

# Run the async workload under test, then assert:
spans = exporter.get_finished_spans()
orphaned = [
    s for s in spans
    if s.parent is None and s.name != "root_span"
]
assert len(orphaned) == 0, f"Orphaned spans detected: {[s.name for s in orphaned]}"

Success criteria: 100% parent-child linkage in test runs, zero parent_id == null anomalies in production monitoring dashboards, and context propagation latency under 50 µs across boundaries.

Related

• Fixing dropped spans in async Python FastAPI routes
• Propagating trace context through Kafka consumers
• When to use tail-based sampling for microservices
• Handling async boundaries in Node.js and Python

↑ Back to Span Lifecycle and Parent-Child Relationships