Trace-Based Alerting and SLO Monitoring

Most teams instrument for tracing first and metrics second, then discover the two signal families were built on separate plumbing that disagrees. The dashboard says the checkout service is at 99.98% availability; the traces show a steady trickle of failed payment calls that never surfaced because the metrics were scraped from a different code path. Trace-based alerting closes that gap by deriving the metrics you page on directly from the same spans that carry your root-cause detail, so an alert and the trace that explains it come from one source of truth.

This guide covers the mechanism end to end: the OpenTelemetry Collector spanmetrics connector that turns spans into RED metrics (Rate, Errors, Duration), how to define latency and error service level objectives and their error budgets from span data, how to write multi-window burn-rate alert rules that page only on genuine budget spend, and how to wire exemplars so every alert deep-links to a representative trace. The child guide on generating RED metrics from spans drills into the connector configuration and PromQL; this page is the full pipeline around it.

Problem Framing

The symptom that drives teams here is an SLO you cannot trust. An availability panel is fed by a metrics exporter bolted onto the HTTP middleware, while errors that originate two hops downstream — a gRPC deadline, a database timeout, a poisoned message on a Kafka consumer — never increment that counter because they never touch the middleware that emits it. Meanwhile the traces record every one of those failures in full. You have two numbers for the same reliability question and they diverge, so the burn-rate alert either never fires or fires on the wrong signal.

Deriving the metric from the span fixes the divergence at its root. Every operation that produces a span produces a data point: a count, an error flag from the span status, and a duration. Aggregating those in the Collector yields RED metrics whose definition of “a request” and “an error” is identical to what you see when you open the trace. The alert and the evidence are two projections of one event stream.

Prerequisites

Before building the pipeline described below, confirm the following are in place:

  • Services export spans over OTLP to a central OpenTelemetry Collector, not directly to a tracing backend. The connector runs inside the Collector, so an intermediate hop is required.
  • Collector version 0.95+ (the spanmetrics connector, which supersedes the deprecated spanmetrics processor, plus exemplar support).
  • Spans set status correctly: server spans that failed carry StatusCode = ERROR, not merely an error log line. Error SLIs read the span status, so an unset status silently under-counts failures. See the SDK setup guide for status conventions.
  • Semantic-convention attributes are populated on server spans: http.request.method, http.route (the templated route, not the raw path), and rpc.grpc.status_code where relevant.
  • A Prometheus-compatible metrics backend (Prometheus, Mimir, or Thanos) started with --enable-feature=exemplar-storage, and a Tempo or Jaeger backend reachable from Grafana for the exemplar deep-link.
  • Agreement from the service owners on the target SLO percentage and the rolling window (28 or 30 days is typical), because the burn-rate thresholds derive arithmetically from those two numbers.

How Spans Become Alertable Signals

The spanmetrics connector sits between a trace receiver and a metrics exporter inside a single Collector process. It reads every span passing through, and for each unique combination of dimensions it maintains a calls_total counter and a duration histogram. Because it runs in the Collector rather than in each SDK, one configuration governs the entire fleet, and the metric definitions stay consistent across languages and frameworks. The diagram below traces one request from raw span to a burn-rate page and back to the exemplar trace that explains it.

Trace-based alerting signal path Spans flow from services into the spanmetrics connector, which emits RED metrics with exemplars to a metrics backend. Recording rules feed a multi-window burn-rate alert. A metric exemplar and the alert both link down to a representative trace in Tempo or Jaeger. Spans OTLP from services spanmetrics connector calls_total + duration histogram + exemplars Metrics backend Prometheus / Mimir recording rules Burn-rate alert multi-window SLO rule RED SLI Exemplar trace Tempo / Jaeger exemplar trace_id drill in on page

The key architectural decision is where the connector reads spans. It must see the full, unsampled stream. If it sits downstream of a head-based or tail-based sampler, your rates and error ratios are scaled by the sampling probability and become meaningless as SLIs. Metrics are cheap to keep at 100%; traces are what you sample. Generate the metrics before the sampler, and let sampling apply only to the branch that writes spans to storage.

Step-by-Step Implementation

Step 1 — Enable the spanmetrics Connector

A connector is both an exporter (for the traces pipeline that feeds it) and a receiver (for the metrics pipeline that reads from it). The pipeline wiring is the part teams most often get wrong, so it is shown in full.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

connectors:
  spanmetrics:
    # Histogram buckets in milliseconds. Choose boundaries near your SLO
    # thresholds so histogram_quantile interpolates accurately around them.
    histogram:
      explicit:
        buckets: [5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2s, 5s]
    # Dimensions become metric labels. Keep them low-cardinality.
    dimensions:
      - name: http.request.method
      - name: http.route          # templated route, NEVER the raw path
      - name: rpc.grpc.status_code
    exemplars:
      enabled: true               # attach trace_id samples to buckets
    # Bound the number of tracked dimension sets to protect memory.
    dimensions_cache_size: 2000
    aggregation_temporality: AGGREGATION_TEMPORALITY_CUMULATIVE
    metrics_flush_interval: 15s

exporters:
  prometheus:
    endpoint: 0.0.0.0:8889
    enable_open_metrics: true     # required to emit exemplars over /metrics

service:
  pipelines:
    # Branch 1: raw spans in -> connector (unsampled, feeds metrics).
    traces:
      receivers: [otlp]
      exporters: [spanmetrics]
    # Branch 2: connector out -> metrics backend.
    metrics/spanmetrics:
      receivers: [spanmetrics]
      exporters: [prometheus]

The connector emits calls_total (a monotonic counter) and a duration histogram, both labelled with service.name, span.name, span.kind, status.code, plus your declared dimensions. Prometheus sanitizes the dots to underscores, so service.name becomes the label service_name in every query below.

Step 2 — Define SLIs as Recording Rules

An SLI (service level indicator) is the measured ratio your SLO targets. Express both the error-ratio and latency SLIs as recording rules over several windows. Precomputing them keeps alert expressions readable and, more importantly, cheap: burn-rate alerts reference the recorded series instead of re-evaluating a heavy rate() over raw histograms on every rule tick.

groups:
  - name: sli-checkout
    interval: 30s
    rules:
      # --- Availability SLI: error ratio over several windows ---
      - record: sli:checkout:error_ratio5m
        expr: |
          sum(rate(calls_total{service_name="checkout", status_code="STATUS_CODE_ERROR"}[5m]))
          /
          sum(rate(calls_total{service_name="checkout"}[5m]))
      - record: sli:checkout:error_ratio1h
        expr: |
          sum(rate(calls_total{service_name="checkout", status_code="STATUS_CODE_ERROR"}[1h]))
          /
          sum(rate(calls_total{service_name="checkout"}[1h]))
      - record: sli:checkout:error_ratio6h
        expr: |
          sum(rate(calls_total{service_name="checkout", status_code="STATUS_CODE_ERROR"}[6h]))
          /
          sum(rate(calls_total{service_name="checkout"}[6h]))
      # --- Latency SLI: fraction of requests slower than the 300ms objective ---
      - record: sli:checkout:slow_ratio5m
        expr: |
          1 - (
            sum(rate(duration_milliseconds_bucket{service_name="checkout", le="250"}[5m]))
            /
            sum(rate(duration_milliseconds_count{service_name="checkout"}[5m]))
          )

The latency SLI here counts the fraction of requests that fall outside the fast bucket, which is the correct shape for a “fast enough” objective. Pick a le boundary that exists in your histogram; interpolating across a missing boundary is where latency SLIs quietly lie.

Step 3 — Compute the Error Budget

The error budget is the arithmetic complement of the SLO over the rolling window. For a 99.9% availability target, the budget is 0.1% of requests; for a 99.5% latency target, 0.5% may be slow. The budget is what burn-rate alerting spends against — you are not paging on “an error happened” but on “we are consuming the month’s allowance too fast.”

# Budget remaining as a fraction (1.0 = full budget, 0 = exhausted),
# for a 99.9% availability SLO over a 30-day window.
1 - (
  sum(increase(calls_total{service_name="checkout", status_code="STATUS_CODE_ERROR"}[30d]))
  /
  sum(increase(calls_total{service_name="checkout"}[30d]))
) / 0.001

Chart this on the service dashboard. A budget that trends toward zero mid-window is the leading indicator that justifies the pages the next step generates.

Step 4 — Author Multi-Window Burn-Rate Alerts

Burn rate is how fast you are spending the budget relative to a steady exhaust-at-window-end pace. A burn rate of 1 exhausts the budget exactly at the window boundary; a burn rate of 14.4 exhausts a 30-day budget in roughly 2 days. The multi-window, multi-burn-rate pattern pairs a fast window (catches acute incidents quickly) with a slow window (suppresses flapping) and requires both to breach before paging.

groups:
  - name: slo-burn-checkout
    rules:
      # Fast burn: 2% of a 30-day budget in 1 hour. Pages the on-call.
      - alert: CheckoutErrorBudgetFastBurn
        expr: |
          sli:checkout:error_ratio1h > (14.4 * 0.001)
          and
          sli:checkout:error_ratio5m > (14.4 * 0.001)
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "Checkout burning error budget 14.4x — ~2 days to exhaustion"
      # Slow burn: 5% of budget in 6 hours. Opens a ticket, no page.
      - alert: CheckoutErrorBudgetSlowBurn
        expr: |
          sli:checkout:error_ratio6h > (6 * 0.001)
          and
          sli:checkout:error_ratio1h > (6 * 0.001)
        for: 15m
        labels:
          severity: ticket
        annotations:
          summary: "Checkout burning error budget 6x over 6h — investigate before it accelerates"

The 0.001 factor is the error budget (1 − 0.999). The 14.4 and 6 multipliers come directly from the standard SLO alerting tables: they are the burn rates that, sustained, consume a defined fraction of the budget over the paired windows. Change only the SLO and the budget; keep the burn-rate multipliers.

Step 5 — Wire Exemplars to Traces

An alert that says “P99 latency breached” without a trace is a starting point, not an answer. Exemplars close the loop. With exemplars.enabled set in Step 1 and --enable-feature=exemplar-storage on Prometheus, each histogram bucket carries occasional trace_id samples. In Grafana, configure the Prometheus data source to expose exemplars and map the trace_id label to your Tempo (or Jaeger) data source:

# Grafana Prometheus data source, exemplar trace-link config
jsonData:
  exemplarTraceIdDestinations:
    - name: trace_id
      datasourceUid: tempo-uid       # the Tempo data source UID
      urlDisplayLabel: "View trace"

Now a latency panel renders exemplars as diamonds along the P99 line; clicking one opens the exact slow trace in Tempo. This is the same correlation described in correlating logs, metrics, and traces, applied specifically to the metric-to-trace jump, and it is what turns a burn-rate page into a two-click root-cause session using critical-path analysis.

Verification

Confirm each stage independently before trusting the alert.

The connector is emitting metrics. Scrape the Collector’s Prometheus endpoint and check for the series:

curl -s localhost:8889/metrics | grep -E '^calls_total|^duration_milliseconds_bucket' | head
# Expect calls_total{...service_name="checkout"...} and duration_milliseconds_bucket{le="250",...}

Rates track real traffic. Drive a known request volume with a load generator and confirm the recorded rate matches, proving the connector sees the unsampled stream:

sum(rate(calls_total{service_name="checkout"}[1m]))
# Should equal your generated RPS, not RPS × sampling_probability.

The alert fires on a synthetic incident. Inject a burst of errors (a feature flag that returns HTTP 500, or a fault-injection sidecar) and watch the fast-burn rule transition to firing within its for window in the Prometheus /alerts view. Then remove the fault and confirm the 5m window clears the alert quickly while the 1h window decays.

Exemplars resolve to traces. Open the latency panel, click an exemplar diamond, and confirm it opens a trace in Tempo whose root span duration is consistent with the bucket it was sampled from. If the link 404s, the trace_id label mapping in the data source is wrong.

Edge Cases and Gotchas

  1. Sampling applied before the connector skews every rate. This is the most damaging mistake. If a tail_sampling or probabilistic_sampler processor runs upstream of the connector, calls_total counts only survivors and your error ratio is computed over a biased denominator. Always branch the pipeline so the connector reads raw spans and sampling applies only to the storage branch. Measure at the Collector, before sampling.

  2. Span-name cardinality from unaggregated operations. The connector labels every metric with span.name. Instrumentations that embed identifiers into the span name (GET /orders/48213, SELECT WHERE id=…) mint a new time series per value and can add millions of series. Enforce templated names at the SDK, or use a transform processor in the Collector to rewrite span names before the connector reads them.

  3. Dimension cardinality multiplies. Total series is roughly the product of the distinct values of every dimension. Adding http.route (200 routes) × http.request.method (5) × status.code (3) × services (40) is already 120,000 series before histogram buckets multiply it further. Add dimensions deliberately, and never add user-, tenant-, or request-scoped attributes as connector dimensions — those belong on spans, not on aggregated metrics.

  4. Counter resets on Collector restart. calls_total is cumulative per Collector process. When a Collector pod restarts, the counter resets to zero. rate() handles single resets correctly, but a crash-looping Collector produces a sawtooth that corrupts the SLI. Alert on Collector restarts separately and treat sustained restart loops as an SLI-integrity incident.

  5. Recording-rule evaluation cost compounds. Each SLI window (5m, 1h, 6h) is a separate rule evaluated every interval across every high-cardinality series. Defining nine windows across fifty services can dominate Prometheus CPU. Record only the windows your alerts actually reference, and scope each recording rule to the service label rather than computing a global aggregate you then filter.

  6. Histogram bucket boundaries must straddle the SLO threshold. histogram_quantile and threshold ratios interpolate linearly within a bucket. If your latency objective is 300ms but the nearest boundaries are 250ms and 500ms, the SLI is a coarse guess across a 250ms band. Place an explicit bucket boundary at (or just above) each latency objective.

Performance and Scale Notes

  • Connector memory is a function of active dimension sets, not span throughput. The dimensions_cache_size caps the number of tracked series; once full, new dimension combinations are dropped from aggregation rather than growing memory unbounded. Size it to your expected cardinality with headroom, and alert on the connector’s own dropped-series metric so a cardinality regression is visible before it silently loses data.
  • Prometheus ingestion scales with series count, not sample rate. Span-derived metrics add series proportional to the dimension product above. Budget roughly 1–3 KB of RAM per active series in Prometheus; 500,000 span-metric series is on the order of 1 GB of head-block memory before considering the WAL and query load. Mimir or Thanos shard this horizontally when a single Prometheus can no longer hold the cardinality.
  • Exemplar storage is bounded and lossy by design. Prometheus keeps a fixed-size exemplar buffer per series and overwrites oldest first. Exemplars are a sampling aid, not a complete index — do not build alerting logic that assumes an exemplar exists for every bucket. For guaranteed retention of the traces behind a spike, pair exemplars with tail-based sampling that keeps every error and latency-outlier trace.
  • Recording rules amortize alert cost. Evaluating the raw error-ratio expression inside each alert rule re-runs the histogram rate() on every tick. Recording the SLI once per interval and referencing the recorded series in the alert cuts alert-evaluation cost by an order of magnitude at scale, which is why Step 2 precedes Step 4.

Troubleshooting FAQ

Why do my span-derived rates drop after I enable sampling?

If the spanmetrics connector runs in the same pipeline as, or downstream of, a sampling processor, it only counts the spans that survived the sampler, so calls_total under-reports true traffic by the sampling factor. Generate metrics from an unsampled pipeline branch before any tail_sampling or probabilistic_sampler processor, then sample the raw spans on a separate branch for storage.

My Prometheus memory exploded after turning on spanmetrics. What happened?

Almost always unbounded label cardinality. A dimension such as a raw URL path, a user ID, or an unaggregated span name creates one time series per distinct value. Restrict dimensions to low-cardinality attributes like http.route, http.request.method, and status.code, cap dimensions_cache_size, and drop or aggregate high-cardinality span names in the Collector before export.

Why does my burn-rate alert flap on and off?

A single-window alert fires on any transient spike. Use the multi-window multi-burn-rate pattern: require both a short window (for example 5m) and a longer window (1h) to exceed the threshold simultaneously. The long window prevents flapping and the short window resets the alert quickly once the incident clears.

How do exemplars connect a metric spike to a specific trace?

With exemplars enabled, the connector attaches a trace_id sample to individual histogram buckets. Prometheus stores these when started with --enable-feature=exemplar-storage, and Grafana renders them as clickable points on the latency graph that deep-link to the trace in Tempo or Jaeger, so a P99 spike leads straight to a representative slow request.

Should SLIs be measured at the service or at the operation level?

Both, at different tiers. Page on a service-level or route-level SLO that maps to a user-facing journey, because that is what customers experience. Keep per-operation SLIs as diagnostic recording rules you break down to during an incident, not as paging alerts, or the on-call rotation drowns in low-signal noise.

Related

↑ Back to Trace-Based Debugging & Signal Correlation