Querying Traces with TraceQL and Jaeger Search

Problem Framing

“Checkout felt slow around 14:20.” That is the entire bug report, and it is typical. Trace storage holds the answer — somewhere in four million traces from that hour — but the default UI workflow of “pick a service, pick an operation, sort by duration, open traces one at a time” is a slow way to find it, and it answers the wrong question. The slowest trace in a window is usually a pathological outlier: a health check that hit a cold cache, a retry storm, a batch job. The trace you want is the one representative of what users experienced.

Query languages exist to close that gap. TraceQL, Tempo’s structural query language, can express “traces where the checkout endpoint took over two seconds and contained a database span over one second and returned 200” — a question no amount of clicking will answer. Jaeger’s search is less expressive but faster to reach for, and knowing which tool answers which question is most of the skill.

Prerequisites

• A trace backend with a query API: Grafana Tempo 2.x for TraceQL, or Jaeger 1.4x+ for tag search. The backend comparison covers the trade-offs between them.
• Spans that follow the semantic conventions — every query below is written against convention attribute names.
• Knowledge of your sampling configuration, because it determines what the query population actually is.

Concept Deep-Dive: Spans, Span Sets, and Traces

The mental model that makes TraceQL click: a query filters spans, groups the survivors into span sets per trace, and then filters traces by properties of those sets.

Three consequences follow from that model, and they explain most confusing query results:

• A query in braces returns spans, not traces. { name = "SELECT orders" } matches every such span; the UI shows the traces containing them, but the match is at span level.
• Conditions joined outside braces are trace-level. { A } && { B } means “this trace contains a span matching A and a span matching B” — not “a span matching both”, which would be { A && B }. That single distinction is the most common source of wrong results.
• Structural operators express ancestry. { A } >> { B } means a span matching A has a descendant matching B; > restricts it to a direct child. This is how you ask “checkout calls the payment service, which then talks to the fraud API”.

The two engines, side by side

Jaeger is the better tool when you know the service and the tag and simply want traces. TraceQL is the tool when the question involves a relationship between spans, or an aggregate.

Step-by-Step Implementation

Step 1 — Scope hard before you filter

Every query should open with the two constraints that let the engine skip data: resource identity and time.

# Always lead with a resource-level filter — it prunes blocks, not just spans.
{ resource.service.name = "checkout-api" && resource.deployment.environment.name = "production" }

Step 2 — Filter the span set that matters

# Slow server spans on one endpoint, last 30 minutes
{ resource.service.name = "checkout-api"
  && span.http.route = "/checkout/{orderId}"
  && span:kind = server
  && span:duration > 2s }

span:duration, span:kind, span:name, and span:status are intrinsics — properties of the span itself rather than attributes — and they are cheaper to evaluate than attribute lookups.

Choosing the threshold matters more than the syntax. Sorting by duration hands you the extreme tail, which is rarely representative; a threshold just above p95 gives you traces that look like what users actually hit.

Step 3 — Add the structural constraint

This is where TraceQL earns its keep. The question “which slow checkouts were slow because of the database, rather than because of a downstream service” is one query:

# Traces where a slow checkout span has a descendant database span over 1s
{ resource.service.name = "checkout-api" && span:duration > 2s }
  >> { span.db.system.name = "postgresql" && span:duration > 1s }

And the inverse — slow checkouts where the database was not the cause:

{ resource.service.name = "checkout-api" && span:duration > 2s }
  && !{ span.db.system.name = "postgresql" && span:duration > 1s }

The diagram below shows what each operator asserts about the same trace tree, which is the quickest way to remember the difference between >>, >, and a bare &&.

Absence queries are how you find broken instrumentation too: a checkout trace that contains no payment span at all means either the call never happened or the context did not propagate.

Step 4 — Aggregate to prove it is systematic

One trace is an anecdote. Aggregation turns the finding into evidence.

# Which route contributes the most slow spans?
{ resource.service.name = "checkout-api" && span:duration > 2s }
  | by(span.http.route) | count()

# Average latency of the database span, grouped by table
{ span.db.system.name = "postgresql" }
  | by(span.db.collection.name) | avg(span:duration)

# Rate of error spans per service — the RED "E" straight from traces
{ span:status = error } | by(resource.service.name) | rate()

Two aggregate patterns are worth memorising because they answer questions people ask constantly.

“Did this get worse after the deploy?” Group by the version attribute rather than eyeballing two dashboards:

# Latency by build — the regression shows as a step between two versions
{ resource.service.name = "checkout-api" && span:kind = server }
  | by(resource.service.version) | quantile_over_time(span:duration, 0.95)

“Which dependency is dragging the endpoint down?” Aggregate the children of the slow parent rather than the parent itself:

# Average duration of every downstream call made during slow checkouts
{ resource.service.name = "checkout-api" && span:duration > 2s }
  >> { span:kind = client }
  | by(span.server.address) | avg(span:duration)

The second pattern is the query form of critical path analysis: instead of opening one trace and reading the waterfall by eye, you ask the store to rank the dependencies across every slow trace in the window. It is dramatically more reliable, because a single trace may have been slow for an idiosyncratic reason while the aggregate reflects the population.

A practical caution on aggregates: they read every span that survived the filter, so an unbounded by() on a high-cardinality attribute is both slow and unreadable. Group by things with tens of values — route, table, peer address, version — not by trace ID or user ID.

Step 5 — The equivalent in Jaeger

Jaeger’s syntax is flatter. Tags use a simple key=value list, and duration bounds apply to the whole trace:

# Jaeger UI search fields
Service:    checkout-api
Operation:  GET /checkout/{orderId}
Tags:       http.response.status_code=500 error=true
Min Dur:    2s
Lookback:   1h

The same query through the HTTP API, which is what you want in a runbook:

# Jaeger query API — tags are a URL-encoded JSON object
curl -s -G "http://jaeger-query:16686/api/traces" \
  --data-urlencode 'service=checkout-api' \
  --data-urlencode 'operation=GET /checkout/{orderId}' \
  --data-urlencode 'tags={"http.response.status_code":"500"}' \
  --data-urlencode 'start='$(( ($(date +%s) - 3600) * 1000000 )) \
  --data-urlencode 'limit=20' \
  | jq -r '.data[] | "\(.traceID)  \(.spans | length) spans"'

Because Jaeger cannot express relationships, the working pattern is: filter as tightly as the tag index allows, then post-process the returned JSON:

# Find returned traces that contain a slow database span — the relationship
# that Jaeger cannot express, done client-side over a narrow result set.
curl -s -G "http://jaeger-query:16686/api/traces" \
  --data-urlencode 'service=checkout-api' --data-urlencode 'limit=200' \
  | jq -r '.data[] | select(any(.spans[];
      .duration > 1000000 and any(.tags[]; .key == "db.system.name")))
      | .traceID'

Verification

A query is doing what you think when three things hold:

• The result count is plausible. Hundreds of matches from a 30-minute window on a 500 RPS service is reasonable; four million is a missing filter, and zero is usually a typo in an attribute name — span.http.route and resource.service.name live in different namespaces and are not interchangeable.
• Opening a result confirms the shape. The trace should visibly contain the spans your query claimed.
• The aggregate matches an independent source. Compare the count from | rate() against the equivalent RED metric. A systematic gap points at sampling, not at the query.

Keep the queries, not just the answers

The queries that matter get re-derived from scratch during every incident, badly, by whoever is on call. Storing them alongside the service is a small investment with a disproportionate return.

A useful runbook entry is three lines: the question in plain English, the query, and what a healthy answer looks like. “Are checkouts slow because of the ledger database? — { resource.service.name = "checkout-api" && span:duration > 2s } >> { span.db.collection.name = "ledger" } — fewer than ten matches per hour is normal.” The last clause is what makes it usable at three in the morning by someone who has never run it before, because it converts a number into a judgement.

Two conventions keep such a collection healthy. First, parameterise the time range rather than hard-coding it, so the same query serves both a live incident and a post-mortem over last Tuesday. Second, review the collection whenever attribute names change — a query written against http.method silently returns nothing once the fleet migrates to http.request.method, and a query that returns nothing looks exactly like a healthy system. Pinning the semantic conventions across services is what keeps a saved query working for more than a quarter.

Finally, promote anything you run repeatedly into a span metric. A TraceQL aggregate executed every thirty seconds by a dashboard is expensive and competes with interactive queries; the same number computed once by the span-metrics connector is nearly free to read and survives the retention window of your metrics store rather than that of your traces.

Edge Cases and Gotchas

1. Attribute namespaces are not interchangeable. span.foo, resource.foo, and bare .foo search different scopes; the bare form searches both and costs more.
2. String comparison is exact and case-sensitive. span.http.route = "/Checkout" will not match /checkout. Use =~ for regex when the value shape is uncertain, and accept the scan cost.
3. Duration comparisons need units. span:duration > 2 is invalid; write 2s, 500ms, or 1500us.
4. Sampling silently changes the population. Search sees stored traces only. With a 1% head-based sample, a query returning three traces represents roughly three hundred requests.
5. Tempo’s block index only covers dedicated columns. Attributes stored in the generic key/value column require a scan; if a query is central to your runbooks, promote its attribute to a dedicated column in the block config.
6. Jaeger’s tag search hits an index cap. With Elasticsearch, high-cardinality tags can exceed field limits and stop being indexed at all, so queries on them quietly return nothing.

Performance and Scale Notes

Time range is the dominant cost. Doubling the window doubles the blocks scanned. Start at fifteen minutes around the reported incident and widen only if empty.

Resource filters prune blocks; span filters do not. Tempo stores block-level summaries of resource attributes, so resource.service.name eliminates entire files. A span-attribute filter still requires decompressing what remains.

Aggregations scan everything that matched. | count() over a wide window is orders of magnitude more expensive than fetching twenty traces. Prototype on a narrow window, then widen.

Concurrent queries compete for the same queriers. A dashboard panel running an expensive TraceQL aggregate every 30 seconds can starve interactive debugging during an incident — which is when you need it most. Precompute recurring aggregates as span metrics instead, as covered in generating RED metrics from spans.

Troubleshooting FAQ

Why does my TraceQL query time out?

It is scanning blocks nothing can prune. Add resource.service.name and tighten the time range; those two constraints do more than any number of span-level conditions.

What is the difference between a span filter and a trace filter?

Braces select spans; operators between brace groups select traces. { A && B } needs one span matching both; { A } && { B } needs two spans in the same trace.

Can Jaeger search do the same structural queries as TraceQL?

No. Jaeger filters by service, operation, tags, and duration, with no way to relate two spans. Filter narrowly, then post-process the JSON — or run that question in Tempo.

How do I find traces that are missing a span?

Negate a span set: { span.http.route = "/checkout" } && !{ resource.service.name = "payment-api" } returns checkouts where the payment service never appeared.

Why do my searches return fewer traces than the metrics suggest?

Sampling. Metrics cover every request; search covers stored traces. Compare rates, not counts.

Related

• Writing TraceQL queries for latency outliers — percentile-driven query patterns that find the representative slow trace
• Finding error traces across services in Jaeger — tag search, error taxonomies, and the API for runbooks
• Finding latency bottlenecks with critical path analysis — what to do once the query returns a trace
• Diagnosing missing and broken traces — when the query returns nothing because the data was never stored
• Correlating logs, metrics, and traces — pivoting from a matched trace to the other two signals

↑ Back to Trace Debugging & Signal Correlation