Generating RED Metrics from Spans

Q: Should I use span.name or http.route as the operation label?

Use the templated http.route (/orders/{id}) as the operation dimension and keep span.name aggregated. Raw paths that embed identifiers create one time series per value and blow up cardinality, which is the single most common cause of a Prometheus outage after enabling span metrics.

Point the OpenTelemetry spanmetrics connector at your unsampled span stream and it emits calls_total and a duration histogram per service and operation — the three PromQL queries below then read Rate, Errors, and Duration straight out of those two series.

Context and When It Matters

RED — Rate, Errors, Duration — is the minimal metric set that describes the health of a request-driven service. You could instrument all three by hand in every handler, but you already emit a span for each operation, and that span already carries a start time, an end time, and a status. Deriving RED from spans means the metrics and the distributed traces agree by construction, with no second instrumentation path to drift out of sync.

This matters most when you want one definition of “a request” across a polyglot fleet. Language-specific metric libraries count requests slightly differently in each framework; the spanmetrics connector counts spans identically everywhere, in one place in the OpenTelemetry Collector. It also removes a whole class of instrumentation drift: there is no separate metrics middleware to forget on a new endpoint, no counter that increments on the happy path but not on the exception path. If the operation produced a span, it produced a RED data point.

This page is the focused how-to for the connector output and the queries. For turning these metrics into SLOs and burn-rate alerts, see the parent guide on trace-based alerting and SLO monitoring.

The Minimal Working Configuration

Start with the smallest connector setup that produces usable RED series, then read the explanation below it:

connectors:
  spanmetrics:
    histogram:
      explicit:
        buckets: [5ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s]
    dimensions:
      - name: http.route            # operation label (templated, low cardinality)
      - name: http.request.method
    exemplars:
      enabled: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [spanmetrics]       # spans flow OUT to the connector
    metrics/red:
      receivers: [spanmetrics]       # metrics flow IN from the connector
      exporters: [prometheus]

The connector reads every span in the traces pipeline and produces two metric families, both automatically labelled with service.name, span.name, span.kind, and status.code:

calls_total — a cumulative counter incremented once per span. This is the raw material for Rate and Errors.
duration_milliseconds — a histogram of span durations, exposed as duration_milliseconds_bucket, duration_milliseconds_sum, and duration_milliseconds_count. This is the raw material for Duration percentiles.

Prometheus rewrites the dotted attribute names to underscores on ingest, so service.name is queried as the label service_name, and the error status appears as status_code="STATUS_CODE_ERROR".

Implementation

The block below is the production-shaped configuration, annotated line by line with what each choice does to the resulting series. The example service is an orders-api exposing gRPC and HTTP.

connectors:
  spanmetrics:
    # Histogram boundaries in milliseconds. Every boundary is one extra
    # time series per label set, so keep the list short and place a bucket
    # near each latency threshold you will later query with histogram_quantile.
    histogram:
      explicit:
        buckets: [5ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s]

    # Dimensions become metric LABELS. Total series ~= product of distinct
    # values across all dimensions x buckets. Only add low-cardinality keys.
    dimensions:
      - name: http.route             # "/orders/{id}" — templated, NOT the raw path
      - name: http.request.method     # a fixed, small set: GET, POST, ...
      - name: rpc.grpc.status_code    # bounded gRPC status enum

    # Attach trace_id samples to buckets so a latency panel can deep-link
    # to a representative trace. Requires an OpenMetrics-capable exporter.
    exemplars:
      enabled: true

    # Hard ceiling on tracked label sets. When full, new combinations are
    # dropped from aggregation instead of growing Collector memory forever.
    dimensions_cache_size: 1500

    # Cumulative counters play correctly with rate(); Prometheus expects them.
    aggregation_temporality: AGGREGATION_TEMPORALITY_CUMULATIVE

exporters:
  prometheus:
    endpoint: 0.0.0.0:8889
    enable_open_metrics: true         # OpenMetrics format is required for exemplars

service:
  pipelines:
    # The connector MUST see the full stream. Put any probabilistic_sampler
    # or tail_sampling processor on a SEPARATE traces branch that exports to
    # storage, never on this branch, or Rate and Errors are scaled by the
    # sampling probability and stop meaning anything.
    traces:
      receivers: [otlp]
      exporters: [spanmetrics]
    metrics/red:
      receivers: [spanmetrics]
      exporters: [prometheus]

With that running, the three RED signals are read directly. Rate is the request throughput per operation:

# Requests per second for orders-api, broken down by operation.
# Filter to server spans so the client span of the same call is not double-counted.
sum by (http_route) (
  rate(calls_total{service_name="orders-api", span_kind="SPAN_KIND_SERVER"}[5m])
)

Errors is the fraction of those requests that failed, computed from the same counter with a status_code filter on the numerator:

# Error ratio (0-1) per operation: errored server spans / all server spans.
sum by (http_route) (
  rate(calls_total{service_name="orders-api", span_kind="SPAN_KIND_SERVER", status_code="STATUS_CODE_ERROR"}[5m])
)
/
sum by (http_route) (
  rate(calls_total{service_name="orders-api", span_kind="SPAN_KIND_SERVER"}[5m])
)

Duration is a latency percentile from the histogram — note that le must be preserved in the by() clause for histogram_quantile to work:

# P99 latency in milliseconds per operation.
histogram_quantile(
  0.99,
  sum by (http_route, le) (
    rate(duration_milliseconds_bucket{service_name="orders-api", span_kind="SPAN_KIND_SERVER"}[5m])
  )
)

Filtering to span_kind="SPAN_KIND_SERVER" is deliberate and important: a single remote call produces both a client span (in the caller) and a server span (in the callee), and counting both doubles your rate. Pick one kind — server for inbound-traffic SLIs, client when you want to measure a dependency as the caller experiences it — and hold it consistent across all three queries.

When you need a cheap average rather than a percentile, divide the histogram _sum by its _count instead of running histogram_quantile:

# Mean latency (ms) per operation — useful on overview panels where a
# percentile would be too expensive to compute across every operation.
sum by (http_route) (rate(duration_milliseconds_sum{service_name="orders-api", span_kind="SPAN_KIND_SERVER"}[5m]))
/
sum by (http_route) (rate(duration_milliseconds_count{service_name="orders-api", span_kind="SPAN_KIND_SERVER"}[5m]))

Reach for the average only as a summary. Averages hide the tail that traces exist to expose, so any latency SLI you page on should use histogram_quantile against the buckets, with a boundary placed at the objective. The average query is fine for a fleet-wide overview panel where computing a percentile per operation would be prohibitively expensive.

One subtlety governs all three queries: aggregation_temporality. Cumulative counters (the configuration above) never reset except on a Collector restart, which rate() tolerates. If you switch the connector to delta temporality — sometimes done to reduce backend storage — rate() no longer applies and you must aggregate the deltas differently. Unless a specific backend requires delta, keep cumulative; it is what Prometheus and the queries here assume.

Verification Checklist

Before wiring these series into dashboards or alerts, confirm:

curl localhost:8889/metrics | grep calls_total returns series with the expected service_name and http_route labels.
The Rate query equals your load generator’s known RPS, proving the connector reads the unsampled stream and no sampler sits upstream.
duration_milliseconds_count and calls_total match for the same label set (they increment together; a mismatch means the two queries aggregate over different labels).
A deliberately failed request increments calls_total{status_code="STATUS_CODE_ERROR"} and moves the error-ratio query above zero.
http_route shows templated values (/orders/{id}), not raw paths with embedded IDs — the fastest cardinality check there is.

Common Pitfalls

Unbounded dimensions. Adding a high-cardinality attribute — user ID, tenant ID, raw path, request ID — as a connector dimension mints one time series per distinct value and can add millions of series in minutes. Keep dimensions to bounded enums (http.route, http.request.method, status.code) and cap dimensions_cache_size. Request- or tenant-scoped data belongs on the span, never on an aggregated metric label.
http.route versus the raw path. If instrumentation records span.name or a path attribute as GET /orders/48213 instead of the templated GET /orders/{id}, every order ID becomes its own series. Fix it at the SDK by using framework route instrumentation, or rewrite names with a Collector transform processor before the connector reads them. The distinction between templated route and raw path mirrors the attribute-hygiene concerns in security boundaries in distributed tracing.
Double-counting under retries. calls_total counts span attempts, not logical requests. A client that retries a failed call three times produces three server spans, so a single user-perceived request inflates both Rate and Errors. This is usually acceptable for health signals, but if you need per-logical-request accounting, tag retried spans with a distinct attribute at the SDK and exclude them, and never sum client and server spans together.

FAQ

Does calls_total count logical requests or span attempts?

It counts spans. A client that retries a failed call three times emits three client spans and the server emits up to three server spans, so calls_total increments per attempt, not per logical request. Compute Rate and Errors from server spans of one span.kind to avoid double-counting the client-plus-server pair, and be aware that retries still inflate the count.

Why is duration_milliseconds_count different from calls_total?

They should match per label set, because the histogram increments its _count on every observed span just as the counter does. If they diverge, you are aggregating over different label sets in the two queries, usually because one query includes a dimension the other drops. Align the by() clauses before treating the divergence as a bug.

Should I use span.name or http.route as the operation label?

Use the templated http.route (/orders/{id}) as the operation dimension and keep span.name aggregated. Raw paths that embed identifiers create one time series per value and blow up cardinality, which is the single most common cause of a Prometheus outage after enabling span metrics.

Related

Trace-Based Alerting and SLO Monitoring — turn these RED series into SLIs, error budgets, and burn-rate alerts.
Correlating Logs, Metrics, and Traces — connect the metric back to logs and the exemplar trace.
OpenTelemetry Collector Pipeline Configuration — the receiver and pipeline wiring the connector plugs into.

↑ Back to Trace-Based Alerting and SLO Monitoring