Span Attributes and Semantic Conventions

Problem Framing

Two services in the same request path describe the same HTTP call three different ways. The Node.js gateway writes http.method and http.url. The Python service that replaced it last quarter writes http.request.method and url.full. The hand-instrumented Go worker in between writes method and endpoint because whoever added the span picked names that read well in the code review. Every trace still renders in Jaeger, so nothing looks broken — until someone asks a question that spans all three services.

“Show me every 5xx on the checkout route across the fleet” cannot be answered, because there is no single key to filter on. The backend’s automatic service map has holes where it could not infer the peer service. The RED dashboards that Grafana generates from span metrics cover the two services that happen to use the convention and silently omit the third. Latency analysis groups by http.route and reports one enormous bucket labelled unknown.

Semantic conventions exist to make that class of question answerable. They are the agreed vocabulary — key names, value types, and value shapes — that turns a bag of free-form key/value pairs into telemetry a machine can reason about. This page covers what the conventions specify, where each attribute belongs, how to migrate a fleet that has drifted, and how to keep the vocabulary from exploding your storage bill through unbounded cardinality.

Prerequisites

• An OpenTelemetry SDK already emitting spans from at least one service (SDK 1.20+ for the stable HTTP conventions).
• The semantic conventions package for your language installed alongside the SDK — opentelemetry-semantic-conventions (Python/JS) or io.opentelemetry.semconv:opentelemetry-semconv (Java).
• An OpenTelemetry Collector in the path, if you want to normalize attributes centrally rather than redeploying every service.
• Familiarity with the span lifecycle — attributes can only be set while a span is recording.

Concept Deep-Dive: Four Places an Attribute Can Live

The first thing to get right is not what to name an attribute but where to attach it. OpenTelemetry offers four scopes, and choosing the wrong one is the most common and most expensive mistake in an instrumentation review. The scope determines how often the value is written, how the backend indexes it, and what a single wrong value costs you.

Resource attributes identify the entity producing telemetry. They are resolved once, when the SDK builds its Resource, and are attached to every span, metric, and log the process exports. service.name is the only genuinely required one — a backend that receives spans without it groups them under unknown_service, which destroys the service map. Because the resource is the unit backends use to count “how many things am I monitoring”, putting a per-request value there (a tenant ID, a request ID) inflates the resource count into the millions and is the single fastest way to hit a vendor cardinality limit.

Span attributes describe one operation: which route was hit, which table was queried, which status came back. This is where the HTTP, database, messaging, and RPC conventions live, and it is where nearly all of your instrumentation effort belongs.

Span events are timestamped annotations inside a span, each with their own attributes. Exceptions are the canonical case: span.record_exception() writes an event named exception carrying exception.type, exception.message, and exception.stacktrace. Events are stored with the span but are usually not indexed for search, so an event attribute is good for forensics and bad for filtering.

Span links attach a reference to another trace — the batch-processing and fan-out case, covered in depth in span links for batch and fan-out workloads.

Stability levels are a contract, not a suggestion

Every attribute in the registry carries a stability level. Stable keys will not be renamed or repurposed within a major version — http.request.method, http.response.status_code, url.path, server.address, error.type, and the db.* core set reached stability across 2023–2024. Development (formerly experimental) keys can change in any minor release; much of the GenAI, CI/CD, and cloud-provider registry still sits here.

The practical rule: a stable key can be written directly by your services and relied on by dashboards. A development key should be written by your services but normalized in the Collector, so that when it is renamed upstream you change one processor config instead of redeploying forty services.

The keys that actually matter

Most teams need a small subset. These are the ones that unlock backend features — service maps, RED metrics, latency-by-route analysis — rather than merely decorating a waterfall:

http.route deserves special attention because it is the attribute teams most often get wrong. It must be the route template — /orders/{orderId} — not the resolved path /orders/8fa21c. Auto-instrumentation reads the template from the framework router, which is exactly why hand-rolled middleware that sets http.route from request.path produces a cardinality explosion and useless dashboards.

Step-by-Step Implementation

Step 1 — Pin the conventions package and read the stability level

Never type attribute keys as string literals. Every language ships a generated constants package that encodes both the key name and its stability, so a rename surfaces as a compile or import error rather than a silently empty dashboard.

# requirements.txt — pin the semconv package alongside the SDK
# opentelemetry-sdk==1.29.0
# opentelemetry-semantic-conventions==0.50b0

# Stable attributes live in the top-level module and are safe to hard-code against.
from opentelemetry.semconv.attributes import (
    http_attributes,      # http.request.method, http.response.status_code
    url_attributes,       # url.path, url.full, url.scheme
    server_attributes,    # server.address, server.port
    error_attributes,     # error.type
)
# Development-stage attributes are quarantined in the _incubating module.
# Importing from here is a signal: normalize these keys in the Collector.
from opentelemetry.semconv._incubating.attributes import (
    deployment_attributes,   # deployment.environment.name
)

span.set_attribute(http_attributes.HTTP_REQUEST_METHOD, "POST")
span.set_attribute(http_attributes.HTTP_RESPONSE_STATUS_CODE, 201)
span.set_attribute(url_attributes.URL_PATH, "/orders")

Step 2 — Set resource attributes once at SDK initialization

Resource attributes are resolved at startup and never change. Build them from environment variables the deployment system already sets, and let the resource detectors fill in host and container identity.

from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
import os

resource = Resource.create({
    # The one attribute that must never be missing.
    "service.name":                  os.environ["OTEL_SERVICE_NAME"],
    # Version lets you attribute a latency regression to a specific deploy.
    "service.version":               os.environ.get("GIT_SHA", "dev"),
    # Keeps staging spans out of production dashboards in shared storage.
    "deployment.environment.name":   os.environ.get("ENVIRONMENT", "local"),
    # Instance identity — bounded by replica count, not by request volume.
    "service.instance.id":           os.environ.get("HOSTNAME", "unknown"),
})

provider = TracerProvider(resource=resource)

The equivalent without touching code, which is preferable for a fleet you do not want to redeploy:

# The SDK reads these automatically; OTEL_RESOURCE_ATTRIBUTES takes a CSV of key=value
export OTEL_SERVICE_NAME="checkout-api"
export OTEL_RESOURCE_ATTRIBUTES="service.version=$GIT_SHA,deployment.environment.name=production"

Step 3 — Apply the HTTP conventions on server and client spans

A server span describes the request this process handled; a client span describes a request this process made. The two use overlapping keys with different required sets, and span.kind is what tells the backend which side of the edge it is looking at.

// Express middleware — the server side of the HTTP convention.
const { trace, SpanStatusCode, SpanKind } = require('@opentelemetry/api');

app.use((req, res, next) => {
  const span = trace.getActiveSpan();
  if (!span) return next();

  // Required on every HTTP server span.
  span.setAttribute('http.request.method', req.method);
  span.setAttribute('url.path', req.path);
  span.setAttribute('url.scheme', req.protocol);
  // server.address is the host this process serves, not the upstream peer.
  span.setAttribute('server.address', req.hostname);

  res.on('finish', () => {
    span.setAttribute('http.response.status_code', res.statusCode);
    // http.route comes from the matched router layer — the TEMPLATE, never req.path.
    // req.route.path is "/orders/:id"; req.path would be "/orders/8fa21c".
    if (req.route?.path) span.setAttribute('http.route', req.route.path);
    if (res.statusCode >= 500) {
      // error.type is a coarse taxonomy value, not a stack trace.
      span.setAttribute('error.type', String(res.statusCode));
      span.setStatus({ code: SpanStatusCode.ERROR });
    }
  });
  next();
});

Span status and error.type are related but distinct: the status drives the red/green colour in the waterfall and the error-rate numerator, while error.type is the dimension you group by when asking which failure dominates. Set both, and see recording exceptions and error status on spans for the full treatment.

Step 4 — Apply the database and messaging conventions

Database spans are where sanitization and convention intersect. db.query.text is enormously useful for finding the slow query and enormously dangerous if it carries literal values from user input.

# Database client span — db.* convention with a parameterized statement.
with tracer.start_as_current_span("SELECT orders", kind=SpanKind.CLIENT) as span:
    span.set_attribute("db.system.name", "postgresql")
    span.set_attribute("db.namespace", "orders_prod")        # database/schema name
    span.set_attribute("db.collection.name", "orders")       # table
    span.set_attribute("db.operation.name", "SELECT")
    # Store the PARAMETERIZED statement. Literal values here are a data-leak
    # vector and destroy the grouping that slow-query analysis depends on.
    span.set_attribute("db.query.text", "SELECT * FROM orders WHERE tenant_id = $1")
    span.set_attribute("server.address", "orders-db.internal")
    span.set_attribute("server.port", 5432)
    rows = cursor.execute(sql, (tenant_id,))

Messaging spans carry messaging.system, messaging.operation.name (publish, receive, process), and messaging.destination.name. Their real job is letting the backend join a producer to its consumers, which matters most when context crosses a broker rather than an HTTP hop.

Step 5 — Enforce the convention in the Collector

Normalizing centrally is what makes a fleet-wide migration tractable: legacy services keep emitting their old keys, and the Collector rewrites them on the way through. This is the same processor stage used for filtering and transforming spans.

# collector-config.yaml — normalize legacy keys, then drop what nobody queries
processors:
  transform/semconv:
    error_mode: ignore
    trace_statements:
      - context: span
        statements:
          # Migrate the pre-1.21 HTTP keys to the stable names.
          - set(attributes["http.request.method"], attributes["http.method"])
              where attributes["http.method"] != nil
          - delete_key(attributes, "http.method")
          - set(attributes["http.response.status_code"], attributes["http.status_code"])
              where attributes["http.status_code"] != nil
          - delete_key(attributes, "http.status_code")
          # Hand-rolled key from the legacy Go worker.
          - set(attributes["http.route"], attributes["endpoint"])
              where attributes["endpoint"] != nil and attributes["http.route"] == nil
          - delete_key(attributes, "endpoint")
          # Cardinality control: a raw path with an embedded id is unbounded.
          - replace_pattern(attributes["url.path"], "/[0-9a-f]{8,}", "/{id}")

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [transform/semconv, batch]
      exporters: [otlp/tempo]

The diagram below shows where each stage of that normalization happens and what each stage is allowed to fix.

Verification

Confirm the convention took effect at three points rather than trusting the code review.

At the wire. Export to a debug exporter and read the actual attribute set on a real request:

# Point one instance at a logging exporter and issue a request
OTEL_TRACES_EXPORTER=console python -m myservice &
curl -s localhost:8080/orders/8fa21c > /dev/null

# Expect the template, not the resolved path:
#   http.route          = /orders/{orderId}
#   url.path            = /orders/8fa21c
#   http.request.method = GET

In the backend. Run a query that only works if the convention is applied — a group-by on http.route should return a bounded list of endpoints, not thousands of unique paths:

# TraceQL — count distinct routes seen in the last hour
{ resource.service.name = "checkout-api" } | by(span.http.route) | count()

If that returns hundreds of rows for a service with twelve endpoints, http.route is being set from the resolved path somewhere. See writing TraceQL queries for latency outliers for the broader query vocabulary.

In the derived views. Open the backend’s service map. Every edge should be labelled with a peer service rather than an IP or a hostname. Missing server.address on client spans is the usual cause of an anonymous edge.

Edge Cases and Gotchas

1. Dual-emit mode doubles your payload. OTEL_SEMCONV_STABILITY_OPT_IN=http/dup makes instrumentation write both the old and new HTTP keys during a migration. It is the right tool for a cutover window, but leaving it on permanently inflates every HTTP span by roughly 120 bytes and confuses any dashboard that does not filter explicitly.
2. Attribute limits truncate silently. The SDK caps attribute count (128 by default) and value length (unlimited by default, but commonly configured to 1024 or 4096). A span that exceeds the count drops attributes without an error; db.query.text on a long statement is usually the first casualty. Raise OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT deliberately rather than discovering the truncation in an incident.
3. Setting attributes after end() is a no-op. The call succeeds and returns cleanly. Any enrichment that depends on the response — status code, payload size, row count — must run before the span ends, which is why the Express example uses the finish event rather than post-response middleware.
4. Resource attributes are not free per span. They are transmitted once per resource per OTLP batch, not once per span, so a large resource set costs little at scale — unless something varies per request and multiplies the number of distinct resources. Then every batch fragments and export overhead grows superlinearly.
5. Sampled-out spans still pay for attribute construction. If a sampler drops the span, the SDK still evaluates the arguments you passed. Building an expensive attribute value — serializing a payload, computing a hash — costs full price on dropped spans. Guard expensive enrichment with span.is_recording().
6. http.route is unavailable before routing. Middleware that runs before the router cannot know the template. Attach it on response, or use the framework’s route-matched hook; auto-instrumentation does the latter, which is one reason it beats hand-rolled middleware. See auto-instrumentation vs manual span creation.

Performance and Scale Notes

The cost of an attribute is not its bytes on the wire — it is what the backend does with it. A key with unbounded values turns into an unbounded index, and if the same key is promoted into span metrics it becomes an unbounded number of time series. The chart below compares four ways of identifying “which endpoint was this” for a service handling 40 million requests a day.

Payload cost. Each attribute costs roughly len(key) + len(value) + 8 bytes in the OTLP protobuf encoding. Twenty attributes averaging 30 bytes adds ~600 bytes per span; at 20,000 spans/second that is 12 MB/s of additional export bandwidth before compression. Attribute discipline is a bandwidth decision as much as a schema decision.

Index cost. Tempo indexes attributes it is told to; Jaeger with Elasticsearch indexes everything by default. On Elasticsearch, a high-cardinality string attribute inflates the inverted index and slows every query on the same index, not just queries touching that field.

Metric promotion is the sharp edge. The span-metrics connector turns span attributes into RED metric dimensions. Promoting http.route yields twelve series per service; promoting url.path yields hundreds of thousands and will take down a Prometheus instance. Full treatment in controlling span attribute cardinality.

Troubleshooting FAQ

Why do my HTTP spans show both http.method and http.request.method?

Two instrumentation layers on different convention versions are writing to the same span, or the SDK is in dual-emit mode (OTEL_SEMCONV_STABILITY_OPT_IN=http/dup). Pin one convention version across every library in the process, then drop the duplicate key in a Collector transform processor until every service has migrated.

Should high-cardinality values like user IDs go on span attributes?

Only if you need to find that exact request later, and never on anything promoted into metrics. A user ID on a span is one indexed value per span; the same key as a metric dimension is one time series per user. For request-scoped values that need to travel between services rather than be searched, use baggage metadata instead.

What is the difference between a resource attribute and a span attribute?

The resource describes the process — set once at startup, identical on every span it emits. A span attribute describes one operation and varies per span. Backends count resources to size your account, so a per-request value in the resource scope is the fastest route to a cardinality limit.

How do I know whether an attribute key is stable or experimental?

The generated package tells you: stable keys live in the main module, development keys in _incubating (Python), incubating (Java), or the ATTR_* experimental exports (JS). Anything you import from the incubating namespace should be normalized in the Collector so a future rename is a config change.

Do semantic conventions matter if I only use one backend?

Yes. Service maps, RED dashboards, latency-by-endpoint views, and error taxonomies are all generated from convention keys. A span with endpoint instead of http.route still appears in the waterfall but silently disappears from every derived view — which is exactly the kind of gap nobody notices until an incident.

Related

• Controlling span attribute cardinality — how to bound attribute values before they reach storage or metrics
• OpenTelemetry semantic conventions for HTTP and RPC spans — the required and recommended key sets, side by side
• Recording exceptions and error status on spans — status codes, error.type, and exception events
• Span lifecycle and parent-child relationships — when a span is still mutable and when it is not
• Security boundaries in distributed tracing — keeping PII out of the attributes you just standardized

↑ Back to Distributed Tracing Fundamentals & Architecture