Controlling Span Attribute Cardinality

Any attribute may be a span attribute for search; only bounded attributes — those with a value space you can enumerate — may become metric dimensions or index columns, and unbounded values must be templated, bucketed, or dropped before they reach a connector.

Context and when it matters

Cardinality problems announce themselves twice. The first time is quiet: a query gets slower, the storage bill rises a little, nobody investigates. The second is loud: Prometheus falls over at 3 a.m. because the span-metrics connector started emitting a time series per user ID after a release added user.id to a span that feeds it.

The asymmetry to internalise is that a span attribute and a metric dimension cost completely different things. On a span, a unique value costs bytes — perhaps 40 of them, once. As a metric dimension, that same unique value costs a permanent time series: memory in the scrape target, a series in the TSDB, a chunk on disk every two hours, forever, whether or not it is ever queried again. That is why a user.id attribute is fine on a span and catastrophic on a metric.

Where cardinality actually hurts

The same attribute costs three different things downstream A single span attribute with one million distinct values fans out to three consumers. Trace storage pays roughly forty bytes per span, a bounded and linear cost. The search index pays a growing inverted index, a sublinear but real cost. The span metrics connector pays one time series per distinct value, an unbounded cost that persists after the traces expire. One attribute · three very different bills user.id on a span 1,000,000 distinct values Trace storage ~40 bytes per span Search index bigger index, slower queries Span metrics 1,000,000 time series acceptable expires with retention watch it slows unrelated queries never do this outlives the traces The attribute is identical in all three paths — what differs is whether the consumer aggregates on it.

Implementation

Template identifiers out of path-like values

Anything containing an ID, a hash, or a UUID is unbounded. Replace the variable part with a placeholder before the value is recorded, and keep the raw value only where you genuinely need to find one request.

import re

# Order matters: most specific pattern first.
_TEMPLATES = [
    (re.compile(r"/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}"), "/{uuid}"),
    (re.compile(r"/\d+"), "/{id}"),
    (re.compile(r"/[0-9a-f]{16,}"), "/{hash}"),
]

def templated_path(path: str) -> str:
    """Turn /orders/8fa21c/items/42 into /orders/{hash}/items/{id}."""
    for pattern, replacement in _TEMPLATES:
        path = pattern.sub(replacement, path)
    return path

# Bounded value for aggregation; raw value kept for forensics only.
span.set_attribute("http.route", templated_path(request.path))
span.set_attribute("url.path", request.path)

Bucket numeric values

A raw byte count, row count, or latency reading has as many values as it has observations. Bucketing converts it into a dimension you can group by.

// Powers-of-ten buckets keep an unbounded number bounded at ~7 values.
function sizeBucket(bytes) {
  if (bytes < 1_000) return 'lt_1kb';
  if (bytes < 10_000) return 'lt_10kb';
  if (bytes < 100_000) return 'lt_100kb';
  if (bytes < 1_000_000) return 'lt_1mb';
  return 'gte_1mb';
}

// Exact value for search; bucket for grouping.
span.setAttribute('http.response.body.size', bytes);
span.setAttribute('response.size.bucket', sizeBucket(bytes));

Allow-list the dimensions a connector may use

This is the control that actually prevents the 3 a.m. incident. The span-metrics connector should name its dimensions explicitly; anything not listed is ignored no matter what the span carries.

# collector-config.yaml — an explicit, bounded dimension set
connectors:
  spanmetrics:
    # Every dimension here multiplies the series count. Four bounded keys
    # across 12 routes, 5 methods, 8 statuses and 20 services is ~38k series.
    dimensions:
      - name: http.route
      - name: http.request.method
      - name: http.response.status_code
      - name: deployment.environment.name
    exclude_dimensions: [span.kind, status.code]
    histogram:
      explicit:
        buckets: [10ms, 50ms, 100ms, 250ms, 500ms, 1s, 2s, 5s]

processors:
  # Defence in depth: strip known-unbounded keys before they reach a connector.
  attributes/prune_for_metrics:
    actions:
      - key: user.id
        action: delete
      - key: session.id
        action: delete
      - key: url.query
        action: delete

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [attributes/prune_for_metrics, batch]
      exporters: [spanmetrics, otlp/tempo]

Do the arithmetic before you add a dimension

Series count is the product of every dimension’s value count, per service. That multiplication is what turns a small decision into an outage:

Series count as dimensions are added, one service Four bars on a logarithmic scale. Route, method and status produce 480 series. Adding deployment environment produces 1,440. Adding pod name produces 288,000. Adding user ID produces 288 million, marked as an outage. Each dimension multiplies · log scale route × method × status 480 series + environment 1,440 series + pod name (200 pods) 288,000 series + user id (1M users) 288M — outage A dimension is affordable only if you can state its value count out loud. If you cannot, it is unbounded.

Keep the raw value where it is still useful

Templating and bucketing are not about deleting information — they are about putting each form of it where it costs the least. A well-instrumented span usually carries both: the bounded value for grouping, the raw value for finding the one request someone is asking about.

That pairing is what lets an investigation move in both directions. You start from an aggregate — “the /orders/{id} route regressed at 14:20” — using the bounded dimension, then filter to individual traces using the raw one — “show me traces where url.path contains this specific order”. Deleting the raw value entirely saves a few bytes and makes the second step impossible; promoting it to a dimension makes the first step expensive. Keeping both, in their proper places, costs almost nothing.

The one case where you should genuinely drop rather than template is data you are not permitted to store. An email address in a path segment is not a cardinality problem, it is a retention and compliance problem, and the answer is redaction at the Collector rather than a clever bucket. The distinction matters because the two look similar in a review and have different fixes — see scrubbing PII from span attributes for that side of it.

A last practical note: cardinality tends to arrive through libraries rather than through your own code. A framework that names spans after the full path, an HTTP client that records the resolved URL as the span name, an ORM that emits the interpolated statement — each adds unbounded values without anyone deciding to. Auditing the distinct-value count per attribute after adding a dependency catches these in minutes, and it is the only reliable way to notice them before the metrics bill does.

Verification

Measure distinct values before a change ships, not after:

# TraceQL — how many distinct values does this attribute really have?
{ resource.service.name = "checkout-api" } | by(span.http.route) | count()

A route dimension returning more rows than you have endpoints means templating is failing somewhere. On the metrics side, the standard health check is series growth:

# Series contributed by the span-metrics connector, by metric name
topk(10, count by (__name__)({__name__=~"traces_span_metrics.*"}))

# Alert when a single metric's series count grows faster than deploys explain
deriv(prometheus_tsdb_head_series[1h]) > 500

Decision criteria

  • Can you enumerate the values? If yes, it can be a dimension. If it depends on user or request volume, it cannot.
  • Does the value grow with traffic? Session IDs, request IDs, and cache keys all do. Keep them on spans, out of connectors.
  • Do you need it for aggregation or for finding one request? Aggregation demands bounded; forensics tolerates unbounded.
  • Is a bucket good enough? Latency, payload size, and row counts are nearly always more useful bucketed than raw.
  • Is it already covered by a bounded key? url.path is unbounded; http.route says the same thing in twelve values.

The same five questions as a flow, for the review checklist:

May this attribute become a dimension? A three-step decision flow. First, can you enumerate the values? If yes it may be a metric dimension. If no, can a template or bucket preserve the meaning? If yes, apply it and use the derived value as the dimension. If no, keep the raw value as a span attribute for search only and exclude it from every connector. Run this before adding any attribute to a connector Can you enumerate the values? Dimension is safe add to the allow-list Does a template or bucket keep meaning? Derive, then aggregate raw stays on the span Search only exclude from connectors yes no yes no

Common pitfalls

  • Adding a dimension “temporarily” during an incident. Series persist for the metrics retention period, long after the incident is closed, and nobody removes them.
  • Templating with a pattern that misses one ID shape. A regex covering numeric IDs but not UUIDs leaves the dimension unbounded while looking fixed. Test against real traffic, then verify with a distinct-count query.
  • Assuming trace storage cardinality limits are the same as metrics. They are not — but a high-cardinality attribute on an Elasticsearch-backed Jaeger index still inflates the index and slows unrelated queries.
  • Treating a low-traffic service as exempt. Series count depends on the value space, not the request rate. A service handling ten requests a minute with a customer.id dimension across fifty thousand customers still produces fifty thousand series; it just takes longer to get there, which makes the cause harder to spot when it finally does.
  • Letting a bucket boundary drift. Buckets defined as “small, medium, large” without explicit thresholds get reinterpreted by the next author, and historical comparisons quietly stop meaning anything. Write the thresholds into the code, not into a comment.
  • Forgetting resource attributes count too. A per-request value in the resource multiplies the resource count, which is worse than a span attribute — see span attributes and semantic conventions.

Related

↑ Back to Span Attributes and Semantic Conventions