Baggage vs Span Attributes: When to Use What

Engineers running distributed tracing across polyglot microservices frequently collide with a concrete failure: a tenant identifier set in the gateway disappears by the time it reaches the payment service, or a UUID written to a span attribute causes a cardinality explosion that crashes a Prometheus instance. Both failures stem from the same confusion — using OpenTelemetry Baggage and Span Attributes interchangeably, when they are architecturally distinct primitives with entirely different propagation scopes, storage models, and performance envelopes.

Problem Framing

The failure mode is specific: a service sets metadata using whichever OpenTelemetry API is closest to hand, but that metadata either never reaches the services that need it, or it does reach them but cannot be queried, or it propagates everywhere including third-party endpoints it should never touch. Symptoms include broken multi-tenant routing, 431 header-too-large errors from API gateways, trace search returning zero results for a field the team is sure was recorded, and compliance alerts from security tooling detecting PII in outbound headers.

The root cause is architectural: Baggage and Span Attributes exist at different layers of the tracing stack. Conflating them is not a subtle SDK misuse — it produces observable production incidents.

Prerequisites

• OpenTelemetry SDK installed and initialised for your language (Python opentelemetry-sdk >= 1.20, Go go.opentelemetry.io/otel >= 1.21, Node.js @opentelemetry/sdk-trace-node >= 1.18)
• At least one tracer provider and exporter configured (see OpenTelemetry SDK setup for backend services)
• Basic familiarity with W3C TraceContext propagation and how traceparent headers flow between services
• (Optional) An OpenTelemetry Collector instance if you intend to enrich spans from baggage at the pipeline level

Concept Deep-Dive: Two Primitives, Two Layers

What OpenTelemetry Baggage Actually Does

OpenTelemetry Baggage implements the W3C Baggage specification, enabling key-value pairs to propagate across HTTP and gRPC boundaries via a standardised baggage header. Unlike trace context, which strictly tracks causal relationships, baggage carries application-defined metadata that must survive across process boundaries, load balancers, and third-party integrations.

Baggage lives in the execution context, not in the span lifecycle. The SDK’s propagator injects it into outbound requests and extracts it from inbound requests before any span creation begins. Because it travels independently of span creation, it can be read by middleware, routers, or authorisation layers without requiring an active trace. This makes baggage the canonical mechanism for scenarios where tenant IDs, feature flags, or routing tokens must traverse service boundaries without coupling to telemetry collection — as covered in the tenant context propagation patterns for multi-tenant SaaS.

The Role of Span Attributes

Span attributes are key-value pairs attached directly to a specific span in the trace graph. They are strictly local to that span’s lifecycle and are not automatically propagated to downstream services. Attributes are optimised for trace backend storage engines — they are indexed, sampled, and exposed to query languages in Jaeger, Tempo, SigNoz, and Datadog.

High-cardinality attributes such as user.id, request.path, and db.statement drive filtering, alerting, and metric derivation. They incur storage costs proportional to cardinality and sampling rates. Unlike baggage, attributes are stripped from the network layer — they exist only in the telemetry pipeline after a span is exported.

Decision Matrix

Step-by-Step Implementation

Step 1 — Register Composite Propagators

Correct propagator registration is mandatory for baggage to survive network hops. The SDK must explicitly register both TraceContextPropagator and W3CBaggagePropagator as a composite in the global configuration. Without this, the baggage header is silently ignored on every inbound request.

Python

from opentelemetry.propagate import set_global_textmap
from opentelemetry.propagators.composite import CompositePropagator
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
from opentelemetry.baggage.propagation import W3CBaggagePropagator

set_global_textmap(CompositePropagator([
    TraceContextTextMapPropagator(),
    W3CBaggagePropagator(),
]))

Go

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/propagation"
)

func init() {
    otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
        propagation.TraceContext{},
        propagation.Baggage{},
    ))
}

Node.js

const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { CompositePropagator, W3CTraceContextPropagator, W3CBaggagePropagator } = require('@opentelemetry/core');

const provider = new NodeTracerProvider();
provider.register({
    propagator: new CompositePropagator({
        propagators: [new W3CTraceContextPropagator(), new W3CBaggagePropagator()],
    }),
});

Step 2 — Set Baggage and Span Attributes Correctly

The scope difference dictates which API to call. Baggage modifies the current execution context; attributes bind to an active span.

from opentelemetry import baggage, context, trace

# Baggage: attaches to execution context, propagates to every downstream call
ctx = baggage.set_baggage("tenant.id", "acme-corp")
token = context.attach(ctx)  # must attach so outbound calls inherit the value

# Span attribute: local to this span only, exported to the trace backend
tracer = trace.get_tracer("order-service")
with tracer.start_as_current_span("process_order") as span:
    span.set_attribute("order.total_usd", 149.99)
    span.set_attribute("db.statement", "SELECT * FROM orders WHERE id = ?")
    # business logic here

context.detach(token)  # always detach; put in finally block in production

Production constraint: Baggage values are URL-encoded and placed in the baggage HTTP header. Exceeding ~8 KB, or including non-ASCII characters, causes truncation or outright rejection by reverse proxies. Validate key length (under 256 characters) and value length (under 4096 characters) at the ingress edge before any value enters the propagation chain.

Step 3 — Extract and Map in Framework Middleware

Middleware interceptors should extract inbound headers early, read baggage values for routing decisions, and also copy relevant keys to the current span as attributes so they appear in trace search. This pattern prevents the common mistake of routing on baggage but having no queryable record of which tenant a given trace served.

from fastapi import Request
from opentelemetry.propagate import extract
from opentelemetry import baggage, context, trace

@app.middleware("http")
async def otel_middleware(request: Request, call_next):
    # Extract both traceparent and baggage from inbound headers
    ctx = extract(request.headers)
    token = context.attach(ctx)
    try:
        tenant = baggage.get_baggage("tenant.id", context=ctx)

        # Copy to span attribute so Jaeger/Tempo can filter by tenant
        span = trace.get_current_span()
        if tenant:
            span.set_attribute("routing.tenant_id", tenant)

        response = await call_next(request)
        return response
    finally:
        context.detach(token)  # runs even if an exception is raised

For async frameworks in Node.js or Python, ensure context.with() or an equivalent wrapper is used when spawning background tasks. Failing to reattach context before an await call silently drops baggage for everything downstream of that boundary — see the async context handling guide for the full pattern.

Step 4 — Propagate Baggage Through Message Queues

Message brokers such as Kafka, SQS, and RabbitMQ do not automatically carry HTTP headers. Baggage must be explicitly serialised into message metadata. Span attributes attached to the producer span are lost unless they are manually written into the message payload or headers before publish. This is covered in depth for Kafka in the context propagation across service meshes guide, but the core pattern is:

import (
    "context"
    "go.opentelemetry.io/otel/baggage"
    "github.com/IBM/sarama"
)

// Producer: write each baggage member to a Kafka record header
func publishWithBaggage(ctx context.Context, msg *sarama.ProducerMessage) {
    b := baggage.FromContext(ctx)
    for _, m := range b.Members() {
        msg.Headers = append(msg.Headers, sarama.RecordHeader{
            Key:   []byte("baggage." + m.Key()),
            Value: []byte(m.Value()),
        })
    }
}

// Consumer: reconstruct context from headers before starting a new span
func consumeWithBaggage(headers []*sarama.RecordHeader) context.Context {
    carrier := make(map[string]string)
    for _, h := range headers {
        carrier[string(h.Key)] = string(h.Value)
    }
    return otel.GetTextMapPropagator().Extract(context.Background(),
        propagation.MapCarrier(carrier))
}

Without the consumer-side Extract call, the trace graph fragments: the producer span exists in one trace and the consumer span starts an entirely new, unrelated trace.

Step 5 — Enrich Spans from Baggage in the OTel Collector

Because baggage is not natively indexed by trace backends, you must copy it to span attributes if you want to query by tenant, session, or feature flag in Jaeger or Tempo. Do this in the OpenTelemetry Collector pipeline rather than in application code to avoid coupling business logic to telemetry concerns.

processors:
  transform/enrich_from_baggage:
    trace_statements:
      - context: span
        statements:
          # Copy the tenant.id baggage value to a queryable span attribute
          - set(attributes["tenant.id"], baggage["tenant.id"])
            where baggage["tenant.id"] != nil
          - set(attributes["feature.flag"], baggage["feature.flag"])
            where baggage["feature.flag"] != nil

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [transform/enrich_from_baggage, batch]
      exporters: [otlp/tempo]

After this processor runs, you can query in Tempo:

{ span.attributes["tenant.id"] = "acme-corp" }

Without this enrichment step, baggage is invisible to every trace search UI.

Verification

After deploying the propagator registration and middleware changes, confirm the flow with curl:

curl -sv \
  -H "baggage: tenant.id=acme-corp, env=prod" \
  -H "traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01" \
  http://localhost:8080/api/order 2>&1 | grep -E "< baggage|< traceparent"

If both headers appear in the downstream service’s inbound log, propagation is working. In Jaeger UI, search for tenant.id=acme-corp — results confirm the Collector enrichment processor is active. If you see no results but the header is present, the processor pipeline is misconfigured or the attribute key is mistyped.

For span lifecycle verification, check that the parent-child relationship is intact: the gateway span should be the root, with Order Service and Payment Service spans as children sharing the same traceId.

Edge Cases and Gotchas

1. Async context loss in Python. asyncio.create_task() does not inherit the current baggage context. Capture the context object before spawning the task and reattach it inside the coroutine with context.attach(ctx) wrapped in try/finally. See fixing dropped spans in async FastAPI routes for the complete fix.

2. Goroutine context loss in Go. context.Context must be passed explicitly to every goroutine. There is no implicit inheritance. Passing the context value through a channel or closure argument is the idiomatic Go solution.

3. Proxy header stripping. Nginx, AWS ALB, Envoy, and Istio may silently strip unknown headers. Add explicit pass-through rules: in Nginx, proxy_set_header baggage $http_baggage;; in Envoy, add baggage to the allowed_headers list on the route or virtual host.

4. Cardinality explosion from misplaced UUIDs. Setting a UUID or database primary key as a span attribute creates one unique label value per request. At scale, this exhausts Prometheus label memory and degrades Tempo query performance. Move unbounded identifiers to baggage if they need to propagate, or record them as log fields rather than span attributes.

5. PII in baggage reaching third-party services. Any third-party HTTP call made while baggage is attached will carry that baggage. If user.email is in baggage, it goes to every payment gateway, analytics endpoint, and feature flag API your service calls. Validate and sanitise baggage at the ingress boundary before attaching to context. For the full secure propagation pattern, see how to safely propagate user IDs via OpenTelemetry baggage.

6. Detach omitted in exception paths. If context.detach(token) is not called when an exception is thrown, the execution context is permanently polluted for that thread or async task slot. Always wrap attach/detach in try/finally.

Performance and Scale Notes

Baggage header size is bounded by your reverse proxy’s header buffer. The HTTP spec has no hard limit but practical limits are 8–16 KB depending on proxy configuration. Keep total baggage under 4 KB by enforcing a maximum of 10–15 short key-value pairs. Reject entries that exceed per-key or per-value size limits at the ingress edge rather than propagating them.

Span attribute cardinality should be bounded by design. The OpenTelemetry semantic conventions exist partly to prevent cardinality accidents: http.method has ~7 possible values; http.url can have millions. Follow the convention mappings and route unbounded values through baggage or log records rather than span attributes.

For the OpenTelemetry Collector transform processor, the set() statements execute for every span passing through the pipeline. Keep the expression list short and add where guards to avoid null-checks failing silently. Under high throughput (>10k spans/second), benchmark the processor CPU overhead in a staging environment before rolling out to production.

Troubleshooting FAQ

Why is my baggage missing in downstream services?

The three most common causes: (1) W3CBaggagePropagator is not in the composite propagator list — check the global textmap registration at SDK startup; (2) a reverse proxy is stripping the baggage header — verify with curl -v and check proxy access logs; (3) the framework middleware calls extract() after span creation, so the baggage is present in the context but not visible to the span’s attribute set.

Can I query baggage values in Jaeger or Tempo?

Not without preprocessing. Baggage is not natively indexed by any trace backend. Add an OpenTelemetry Collector transform processor that copies the baggage keys you care about to span attributes before export. Once that is in place, the enriched attribute is fully queryable: { span.attributes["tenant.id"] = "acme-corp" }.

What happens when baggage exceeds the 8 KB header limit?

API gateways and proxies return 431 Request Header Fields Too Large, which manifests as a sudden spike in 4xx errors from the ingress layer. Enforce a strict key allowlist at the edge — reject any key not on the approved list. Keep individual values under 4096 characters, and hash or compress large payloads before setting them as baggage.

Is it safe to put user emails or session tokens in baggage?

No for PII such as emails. Baggage propagates indiscriminately to every downstream call, including third-party APIs, making it a GDPR and CCPA liability. Use hashed or pseudonymised identifiers — sha256(user.id) — if correlation across services is required. Session tokens are safer in baggage when they are opaque short-lived values, but they must still be validated at each service boundary rather than trusted as-is.

How do I propagate baggage through Kafka without losing the trace graph?

On the producer side, iterate over baggage.FromContext(ctx).Members() and write each member as a Kafka record header (e.g., key baggage.tenant.id, value acme-corp). On the consumer side, reconstruct a MapCarrier from those headers and call otel.GetTextMapPropagator().Extract() before starting any new span. Without the consumer-side extract, the consumer’s spans start a fresh root trace and the causal link to the producer is lost. The context propagation through Kafka consumers page has the complete worked example.

Related

• How to Safely Propagate User IDs via OpenTelemetry Baggage
• Tenant Context Propagation in Multi-Tenant SaaS
• Handling Async Boundaries in Node.js and Python
• Context Propagation Through Kafka Consumers

↑ Back to Baggage & Metadata Routing Workflows