Enforcing tenant isolation in trace data

One tenant must never see another tenant’s spans — enforce it by re-validating tenant.id at the first trusted hop, copying that trusted value into a span attribute at the collector, and hard-scoping storage and queries so isolation does not depend on UI filters alone.

Context and when it matters

In a multi-tenant SaaS, a leaked span is a data breach. Trace payloads routinely carry URLs, SQL fragments, customer identifiers, and error messages — if tenant A can pull up tenant B’s traces, you have exposed B’s operational internals to a competitor. The tenant context propagation that makes traces useful is the same mechanism that, done carelessly, leaks them. Isolation has to be enforced at three layers: what value you trust, where you store it, and who can query it.

The failure most teams ship is trusting the tenant.id a client already placed in baggage. Baggage rides in a plain header across the trust boundary; anyone can set it. Isolation that depends on an unverified header is not isolation. The fix is to re-derive the tenant from an authenticated identity, stamp the trusted value onto spans at a controlled point, and then make the storage layer refuse cross-tenant reads.

Collector transform and tenant-header config first

Start with the enforcement point. This Collector pipeline reads the validated tenant.id baggage value into a span attribute, then routes the export under a per-tenant storage scope. Configuration first, explanation after.

processors:
  # 1. Promote the validated baggage entry to a first-class span
  #    attribute so it is queryable and survives export.
  transform/tenant:
    trace_statements:
      - context: span
        statements:
          # Copy tenant.id from baggage into a stable attribute.
          - set(attributes["tenant.id"], attributes["baggage.tenant.id"])
          # Drop the raw baggage mirror so it cannot leak downstream.
          - delete_key(attributes, "baggage.tenant.id")

  # 2. Fail closed: if tenant.id is absent, tag for quarantine
  #    instead of writing it into a shared tenant space.
  transform/guard:
    trace_statements:
      - context: span
        statements:
          - set(attributes["tenant.id"], "UNSCOPED")
            where attributes["tenant.id"] == nil

exporters:
  # 3. Tempo multi-tenancy: X-Scope-OrgID is the hard boundary.
  #    The header value is the tenant; reads must match it.
  otlphttp/tempo:
    endpoint: http://tempo-distributor:4318
    headers:
      X-Scope-OrgID: "${env:TENANT_SCOPE}"   # set per-tenant collector

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [transform/tenant, transform/guard, batch]
      exporters: [otlphttp/tempo]

The transform/tenant processor moves the tenant identifier out of baggage and into a durable tenant.id span attribute, then deletes the baggage mirror so it cannot propagate further than intended. The transform/guard step fails closed: a span arriving with no tenant is labeled UNSCOPED rather than silently landing in a real tenant’s data. Finally, Tempo’s X-Scope-OrgID header is the storage-level boundary — data written under one org ID is a separate logical stream, and a query without the matching header returns nothing.

Where the trusted value comes from

The collector can only stamp a trusted value if something upstream established one. That happens at the first authenticated hop — your API gateway or edge service — where the session token or mTLS identity is available. There, you overwrite whatever the client claimed:

from opentelemetry import baggage, context
from opentelemetry.trace import get_current_span

def establish_tenant(request, ctx):
    # Authenticate FIRST — the token is the source of truth.
    claims = verify_session(request.headers["authorization"])
    trusted_tenant = claims["tenant_id"]          # server-derived, not client-sent

    # Overwrite any client-supplied baggage tenant.id with the
    # authenticated value. Never merge or trust the incoming one.
    ctx = baggage.set_baggage("tenant.id", trusted_tenant, context=ctx)

    # Stamp the current span immediately so even the entry span
    # carries the validated tenant, not the asserted one.
    get_current_span().set_attribute("tenant.id", trusted_tenant)
    return ctx

Two things make this safe. First, verify_session derives the tenant from the authenticated token, so the value cannot be forged by a caller setting a header. Second, the entry span is stamped right away, closing the window where a root span exists without a tenant tag. From here the trusted tenant.id flows as baggage vs a span attribute: baggage carries it across services, and the collector materializes it onto every span.

Query-time access control

Storage scoping stops cross-tenant writes from mixing; query-time control stops cross-tenant reads. Even with per-tenant X-Scope-OrgID streams, the read path must inject the caller’s own scope, never a scope they can choose. Put a proxy in front of the trace UI that derives X-Scope-OrgID from the authenticated operator’s tenant and strips any client-supplied value:

# Read-path proxy rule (pseudo-config): the tenant scope on a
# query is set from the caller's identity, not their request.
- match: { path: "/api/traces*" }
  set_header:
    # Overwrite, do not append — a client cannot pick their scope.
    X-Scope-OrgID: "{{ auth.tenant_id }}"
  remove_request_header: [ "X-Scope-OrgID-Original" ]

For Jaeger, which lacks Tempo’s native org-ID tenancy, the equivalent is a per-tenant storage namespace (separate Elasticsearch index prefix or Cassandra keyspace) plus a query proxy that constrains every search to the caller’s index and appends a mandatory tenant.id tag filter. In both backends the principle is identical: the tenant scope is a server-derived fact, applied on every read, that the user cannot override.

Verification checklist

Common pitfalls

  • Trusting the client-supplied tenant id. Baggage is caller-controlled. If the collector copies tenant.id straight from an unverified header into a span attribute, an attacker relabels their own traffic as another tenant — or reads it back. Always re-derive from an authenticated identity at the trust boundary before the value is stamped.
  • Cross-tenant leakage via a shared root span. In a shared gateway or a batch job that fans out to multiple tenants, a single root span can accumulate children from several tenants. Enforce a fresh trace or an explicit tenant.id reset per tenant unit of work so no span inherits the wrong scope. See security boundaries in distributed tracing for scrubbing PII on those shared hops.
  • High-cardinality tenant attributes. tenant.id on every span is fine for filtering, but feeding thousands of tenant values into a metrics-generator or a hot index inflates cardinality and cost. Keep isolation at the storage-tenancy layer; do not lean on unbounded label sets to enforce it.

FAQ

Why not trust the tenant.id that clients already put in baggage?

Baggage crosses the trust boundary in a plain header a caller can forge. Treat client-supplied tenant.id as an assertion, not a fact: at the first trusted hop re-derive the tenant from the authenticated session or token, and overwrite the baggage value before any span attribute is stamped from it.

How does Tempo enforce per-tenant separation?

Tempo uses the X-Scope-OrgID header as a hard multi-tenancy boundary. Data written under one org ID is stored and queried in a separate logical stream; a read request must present the matching X-Scope-OrgID or it sees nothing. Set that header at the collector from the validated tenant.id.

Is putting tenant.id on every span a cardinality risk?

It can be if tenant.id feeds a metrics-generator or a high-cardinality index. Keep tenant.id as a span attribute for filtering and isolation, but avoid promoting it to a metric label with thousands of values. Scope isolation belongs at the storage tenancy layer, not in unbounded label sets.

↑ Back to Tenant Context Propagation in Multi-Tenant SaaS