Finding Error Traces Across Services in Jaeger

Search the service that reported the symptom with error=true, then read each returned trace from its deepest error span upward — the deepest span carrying an error.type is the root cause, and everything above it is propagation.

Context and when it matters

An alert fires on the checkout error rate. Checkout is almost certainly not broken; something it depends on is, and the error surfaced where the metric happened to be. Jaeger’s job here is to convert “checkout is failing” into “the inventory service is returning 503 because its database connection pool is exhausted”, and it does that well provided you know its limits.

Those limits matter. Jaeger’s search filters by service, operation, tags, duration, and time. It cannot express a relationship between two spans, so questions like “traces where checkout failed and inventory was slow” have to be answered by post-processing the results. Knowing which half of the work the search engine does saves a lot of clicking.

The error chain

Reading an error chain from the bottom up Four nested error spans. The gateway returns 502, the checkout service records upstream error, the inventory service records 503 service unavailable, and the deepest span is a database connection acquisition failure with error type pool exhausted. The deepest span is annotated as the root cause; the three above it are propagation. Four error spans, one cause gateway · ERROR · error.type = 502 symptom checkout-api · ERROR · error.type = upstream_error propagation inventory-api · ERROR · error.type = 503 propagation db.pool.acquire · ERROR · pool_exhausted root cause Read upward: each layer translated the failure below it into its own vocabulary. A mismatch between two adjacent levels — a timeout reported above as invalid_request — is itself a bug worth filing. Alerting on the gateway's error.type tells you nothing; alerting on the deepest one tells you what to fix.

The searches that work

# Jaeger UI — the first search, always
Service:    checkout-api
Tags:       error=true
Lookback:   1h
Limit:      100

# Narrower, once you have a hypothesis
Service:    inventory-api
Tags:       error=true http.response.status_code=503
Min Dur:    (leave empty — errors are often fast)

Leaving the duration filter empty matters more than it sounds. Failures are frequently faster than successes — a connection refused returns in two milliseconds — so a minimum-duration filter carried over from a latency investigation quietly excludes the traces you want.

Two tag conventions determine whether this works at all. error=true is the legacy Jaeger convention and is still what the UI’s error highlighting keys on; otel.status_code=ERROR is what OpenTelemetry sets. Most Jaeger deployments translate one to the other, but if error search returns nothing while errors clearly exist, that translation is the first thing to check. And error.type needs to be a bounded taxonomy value, as described in recording exceptions and error status on spans — a free-text message there makes grouping impossible.

The API, for anything you do twice

#!/usr/bin/env bash
# error_summary.sh — which failure dominates right now?
JAEGER=${JAEGER:-http://jaeger-query:16686}
SERVICE=${1:-checkout-api}
SINCE_US=$(( ($(date +%s) - 3600) * 1000000 ))

curl -s -G "${JAEGER}/api/traces" \
  --data-urlencode "service=${SERVICE}" \
  --data-urlencode 'tags={"error":"true"}' \
  --data-urlencode "start=${SINCE_US}" \
  --data-urlencode 'limit=500' \
| jq -r '
    .data[].spans[]
    | select(any(.tags[]; .key == "error" and .value == true))
    | {
        svc: (.processID),
        type: ([.tags[] | select(.key == "error.type") | .value] | first // "unset")
      }
    | "\(.svc)\t\(.type)"' \
| sort | uniq -c | sort -rn | head -20

That script answers the question a dashboard cannot: not “how many errors” but “which kind, in which service”. Running it at the start of an incident usually removes half the hypotheses.

Post-processing for what Jaeger cannot ask

Jaeger has no structural operator, so relationships must be evaluated client-side over a bounded result set. The pattern is always the same: filter as tightly as the tag index allows, fetch, then reason over the JSON.

# "Error traces where the inventory service was ALSO slow" — a structural
# question, answered by post-processing rather than by the query engine.
curl -s -G "http://jaeger-query:16686/api/traces" \
  --data-urlencode 'service=checkout-api' \
  --data-urlencode 'tags={"error":"true"}' \
  --data-urlencode 'limit=300' \
| jq -r '
    .data[]
    | select(
        any(.spans[];
            .duration > 1000000
            and any(.tags[]; .key == "peer.service" and .value == "inventory-api"))
      )
    | .traceID'

If you find yourself writing several of these, that is the signal to run the question in Tempo instead, where {A} >> {B} expresses it directly — see querying traces with TraceQL and Jaeger search for the comparison.

What error search cannot tell you

Two blind spots are worth naming, because both produce confident wrong conclusions.

Sampling bias. Search sees stored traces only. With head-based sampling at 1%, an error affecting one request in a thousand appears roughly once every hundred thousand requests, and its absence from search proves nothing. If error investigation matters — and it does — that is the argument for tail sampling, which keeps errors at 100% regardless of the baseline rate.

Errors that were handled. A request that failed, retried, and succeeded shows error spans inside a successful trace. Searching for error=true returns it, which is correct, but counting those traces as failures overstates the user-visible error rate substantially. Filter to server spans, or check the root span’s status, before drawing conclusions about impact.

Three different things people call "the error rate" Three nested counts over one hour. Error spans total 41,000. Traces containing at least one error span total 9,200. Traces whose root span failed total 380. Only the last figure corresponds to requests the user saw fail. One hour · three defensible numbers, one right answer spans marked ERROR — 41,000 includes retries and propagation traces with an error — 9,200 includes requests that recovered failed root spans — 380 what users actually experienced A 24× discrepancy between the middle and bottom rows is normal in a system with retries — and alarming if you quote the wrong one.

The searches above find traces. Turning traces into a diagnosis follows a shape that is worth making explicit, because under pressure people tend to open one trace and generalise from it.

Start at the symptom, not the suspicion. Search the service the alert fired on, even when everyone is confident the problem is elsewhere. The chain leads to the cause; starting at the guess skips the evidence that would have contradicted it.

Open three traces, not one. A single failing trace can be idiosyncratic. Three traces from the same window that share a deepest error span are a pattern; three that do not share one mean you are looking at multiple concurrent problems, which is itself an important finding and is invisible from a single sample.

Check whether the failure is new. Run the same search over the same window a day earlier. An error type that has existed at a low rate for weeks and has now increased is a capacity or dependency story; one that appears for the first time is a deploy story. That distinction usually determines who is called next.

Read the timestamps. The deepest error span’s start time relative to the request tells you whether the failure was immediate — connection refused, invalid input — or followed a wait, which points at timeouts and saturation. Immediate failures and slow failures rarely have the same cause even when they share an error type.

Finally, capture the query that found it in the incident notes. The next occurrence is usually months later, when nobody remembers which tag combination worked, and a recorded query converts a repeat investigation into a lookup.

Making errors findable in the first place

Every search on this page depends on the spans carrying usable error data, and that is an instrumentation property rather than a query one. Three things make the difference between a productive search and an empty one: span status set on the service that actually failed, a bounded error.type value that groups, and enough error traces retained to be found at all.

The third is the one teams forget. At a low head-based sample rate, most error traces were never stored, and no amount of query skill recovers them. If error investigation is part of how you operate — and it usually is — that is the strongest available argument for keeping errors at full retention, whether through tail sampling or a dedicated pipeline.

Error spans by service in one incident hour Error spans by service in one incident hour. gateway 9,100 — symptom; checkout-api 8,800 — propagation; inventory-api 8,600 — propagation; db.pool.acquire 8,400 — cause Error spans by service in one incident hour gateway 9,100 — symptom checkout-api 8,800 — propagation inventory-api 8,600 — propagation db.pool.acquire 8,400 — cause Nearly identical counts across four layers is the signature of one cause propagating, not four failures.

Common pitfalls

  • Carrying a duration filter over from a latency search. Errors are often fast; the filter silently excludes them.
  • Searching the failing service instead of the reporting one. Start where the symptom appeared and follow the chain down; starting at the guess skips the evidence.
  • Treating every error span as a failed request. Retries and propagation inflate the count by an order of magnitude.
  • Free-text values in error.type. Grouping becomes impossible and the summary script above returns one row per occurrence.
  • Concluding from an empty result. With head-based sampling, absence of evidence is close to meaningless for rare errors.

Related

↑ Back to Querying Traces with TraceQL and Jaeger Search