Span Links for Batch and Fan-Out Workloads

When one span is caused by many upstream operations — a consumer processing a batch of 500 Kafka messages from 500 different traces — make it a root span with one link per message rather than picking one arbitrary parent, because parent-child can only express a single causal ancestor.

Context and when it matters

Parent-child is a tree, and a tree has exactly one path from any node to the root. That model fits synchronous request paths perfectly and breaks the moment work is aggregated. A batch consumer pulling 500 messages has 500 legitimate causal ancestors. A scheduled job that processes yesterday’s orders has thousands. A fan-out job that spawns a thousand workers has one ancestor but a thousand descendants that outlive it.

Teams meet this problem and reach for one of two bad answers. The first is to pick the first message’s context as the parent, which produces a trace where 499 unrelated requests appear to be children of one arbitrary user’s checkout — misleading in the UI and actively wrong for any latency analysis. The second is to drop context entirely, which makes the batch work invisible from the request that caused it.

Span links are the third answer: a first-class, many-to-many association that says related to without claiming caused by.

Three ways to model a batch, only one of which is honest Three panels. In the first, the batch span is made a child of message one, so the other messages appear unrelated and one user's trace contains work for five hundred others. In the second, the batch span is a root with no relationships, so the work is invisible from any producing trace. In the third, the batch span is a root with one link per message, associating it with every contributing trace without claiming a single cause. One batch span, 500 producers wrong parent msg 1 trace batch span (500 msgs) msg 2..500 unrelated one user's trace shows everyone else's work no context msg traces batch span (orphan) no path from a request to the work it caused span links msg 1 msg 2 …500 batch span · root + 500 links every producer can reach it no false parentage Links are dashed in every UI that renders them — a deliberate visual distinction from causal parentage.

Links are set at span creation and are immutable afterwards, which is the one constraint that shapes the implementation: you must know the full set of related contexts before you start the span.

Implementation

A batch consumer

from opentelemetry import trace
from opentelemetry.propagate import extract
from opentelemetry.trace import Link, SpanKind

tracer = trace.get_tracer("worker.batch")

def process_batch(messages):
    links = []
    for msg in messages[:128]:            # cap — see the scale note below
        # Each message carries its producer's context in its headers.
        ctx = extract({k: v.decode() for k, v in msg.headers})
        sc = trace.get_current_span(ctx).get_span_context()
        if sc.is_valid:
            # Link attributes travel WITH the link, not with the span, so each
            # one can carry its own message identity.
            links.append(Link(sc, {
                "messaging.message.id": msg.key.decode(),
                "messaging.kafka.offset": msg.offset,
            }))

    # Root span — no parent — but associated with every contributing trace.
    with tracer.start_as_current_span(
            "process orders batch", kind=SpanKind.CONSUMER, links=links) as span:
        span.set_attribute("messaging.system", "kafka")
        span.set_attribute("messaging.operation.name", "process")
        span.set_attribute("messaging.batch.message_count", len(messages))
        for msg in messages:
            handle(msg)                   # child spans nest under the batch span

A fan-out job

Fan-out inverts the relationship: one parent, many children that outlive it. Here parent-child is usually still correct — each worker genuinely was caused by the dispatcher — but a link is the right choice when the workers are asynchronous enough that the dispatcher span has long since ended.

// Dispatcher ends immediately; workers run for minutes. Parenting them to a
// span that closed thirty seconds ago produces a trace whose duration is a lie.
const dispatchCtx = trace.getActiveSpan().spanContext();

for (const shard of shards) {
  await queue.publish(shard, {
    // Carry the dispatcher's context in the payload rather than as a parent.
    linkTraceId: dispatchCtx.traceId,
    linkSpanId: dispatchCtx.spanId,
  });
}

// In the worker:
const link = {
  context: { traceId: job.linkTraceId, spanId: job.linkSpanId, traceFlags: 1 },
  attributes: { 'job.shard': job.shard, 'link.kind': 'dispatched_by' },
};
const span = tracer.startSpan(`process shard ${job.shard}`, { links: [link], root: true });

Choosing between the two

Parent-child or link? A two-question decision. If there is exactly one causal ancestor and the parent's lifetime encloses the child's, use parent-child. If there are many ancestors, or the parent ends long before the child, use a span link and start a new root. Two questions decide it exactly one cause? one upstream operation parent outlives child? lifetimes are nested parent-child the normal case link + new root batch, fan-out, schedule yes yes no no A "no" to either question means parent-child would misrepresent the work — a long-running child of a closed parent inflates its duration.

Links are cheap individually and expensive in bulk. Each carries a 16-byte trace ID, an 8-byte span ID, flags, and whatever attributes you attach — roughly 60–100 bytes. Five hundred links is 30–50 KB on a single span, which is larger than most entire traces.

The SDK enforces a cap (OTEL_SPAN_LINK_COUNT_LIMIT, default 128) and drops the excess silently, so a batch of 500 messages with no explicit cap produces a span linked to an arbitrary 128 of them and no indication that 372 are missing. Cap deliberately and record the true count in an attribute:

MAX_LINKS = 64
links = build_links(messages[:MAX_LINKS])
span.set_attribute("messaging.batch.message_count", len(messages))
span.set_attribute("messaging.batch.linked_count", len(links))   # honest about truncation

Backend support also varies more than for parent-child. Tempo and Jaeger both store links and expose them in the UI; the quality of navigation differs, and some query languages cannot filter on link attributes at all. Verify that your backend renders them before designing a workflow around them — a link nobody can follow is only a storage cost.

Verification

  • Open a producer’s trace and confirm the batch span is reachable from it, rendered as a link rather than nested as a child.
  • Open the batch span and confirm the link count matches linked_count, and that message_count shows the real total.
  • Check span size. A batch span far larger than its siblings means the link cap is too generous.
  • Confirm no false parentage: a user’s checkout trace should never contain another user’s work.

Reading a linked trace during an investigation

Links change how an investigation moves, and it is worth knowing the workflow before you rely on them.

Starting from a producer — a user complains, you open their request — the batch span appears as a link rather than as a nested child. Most UIs render it as a separate, clickable trace reference rather than as part of the waterfall, which is correct: the batch work is not part of what the user waited for. One click takes you to the batch trace, where the child spans show what actually happened to that message among the several hundred processed together.

Starting from the batch — an alert fires because a consumer is failing — the links point the other way, at every trace that contributed a message. That is the direction people underuse. When a batch fails, the question “which upstream requests are affected” has an exact answer, and answering it is the difference between “some orders may not have been emailed” and a specific list.

Two practical constraints shape how useful this is. First, a link points at a trace that may have been sampled out; the reference remains but the target is absent, which reads as a broken link. If linked traces matter to you, that is an argument for sampling producers more generously, or for tail sampling that keeps traces the consumer later cares about. Second, link attributes are where the message identity lives, and not every UI displays them; if yours does not, put the same identity on the batch span as a bounded attribute so it is at least searchable.

Finally, links do not participate in critical-path analysis. A tool computing the critical path of a request will not follow a link into the batch work, which is the correct behaviour — the user did not wait for it — but it means asynchronous work needs its own latency objectives rather than being folded into the request’s.

Span size as links are added, 80 bytes each Span size as links are added, 80 bytes each. no links ≈320 B; 8 links ≈960 B; 64 links ≈5.4 KB; 128 links (SDK cap) ≈10.6 KB Span size as links are added, 80 bytes each no links ≈320 B 8 links ≈960 B 64 links ≈5.4 KB 128 links (SDK cap) ≈10.6 KB A batch span with 128 links is larger than most complete traces — cap deliberately and record the true count.

Common pitfalls

  • Adding links after span creation. The API does not allow it in most SDKs; gather contexts first, then start the span.
  • Linking to an invalid context. An all-zero or malformed context produces a link pointing nowhere. Check is_valid before adding, as in the example above.
  • Using links where parent-child would work. A synchronous call with one cause should be a child; links are harder to navigate and are not counted in critical-path analysis.
  • Forgetting the cap. Silent truncation at 128 is the default, and it makes the batch span quietly incomplete.
  • Assuming links propagate sampling. They do not. A linked trace that was sampled out stays absent, so the link may point at a trace that no longer exists.

Related

↑ Back to Span Lifecycle and Parent-Child Relationships