Writing TraceQL Queries for Latency Outliers

Set the duration threshold just above the endpoint’s p95 rather than sorting by slowest, then narrow structurally — “slow and contains a slow database span” — because the slowest traces in any window are almost always pathological cases that explain nothing about what users experienced.

Context and when it matters

The default investigative move is to sort by duration and open the top result. It is the wrong move often enough to be worth unlearning. The slowest trace in a busy window is typically a request that hit a cold cache during a deploy, a retry storm against a briefly unavailable dependency, or an internal batch job that happens to share the endpoint. Fixing what it shows you fixes something nobody experienced.

What you want is a trace from the population that did degrade: slower than normal, but not so extreme that it represents a different phenomenon. That is the band between roughly the 95th and 99th percentile, and querying for it deliberately is one line of TraceQL.

Anchoring the threshold

Three ways to pick which traces to open A latency distribution curve. Sorting by slowest selects only the extreme right tail beyond the 99.9th percentile, labelled as pathological. A round-number threshold of one second cuts across the middle of the distribution and returns a mixture of healthy and degraded traces. A band between the 95th and 99th percentile is marked as the representative population. Which slice of the distribution should you open? p95 p99 p99.9 query this band pathological healthy traffic Sorting by duration always lands to the right of p99.9 — a population of one-off events with no shared cause. A round-number threshold ignores what "normal" is for this endpoint and mixes both populations together. 
# Read the percentile first, then query the band it defines.
{ resource.service.name = "checkout-api" && span:kind = server
  && span.http.route = "/checkout/{orderId}" }
  | quantile_over_time(span:duration, .95, .99)

# Then open traces from between those two numbers, not above them.
{ resource.service.name = "checkout-api" && span:kind = server
  && span.http.route = "/checkout/{orderId}"
  && span:duration > 1.9s && span:duration < 4.2s }

Narrowing structurally

A duration filter alone still returns a mixed population. Adding a structural condition splits it into hypotheses you can test one at a time.

# Slow AND the database was slow — the "database is the cause" hypothesis
{ resource.service.name = "checkout-api" && span:duration > 1.9s }
  >> { span.db.system.name = "postgresql" && span:duration > 800ms }

# Slow AND the database was NOT slow — everything else
{ resource.service.name = "checkout-api" && span:duration > 1.9s }
  && !{ span.db.system.name = "postgresql" && span:duration > 800ms }

# Slow AND a downstream service was slow — with the peer named
{ resource.service.name = "checkout-api" && span:duration > 1.9s }
  >> { span:kind = client && span:duration > 500ms }
  | by(span.server.address) | count()

Running the first two and comparing counts is often the whole investigation. If 90% of slow traces contain a slow database span, the cause is the database; if 10% do, looking at database dashboards is a detour.

Comparing against a healthy baseline

The single most useful habit is to run every diagnostic query twice — once against the degraded population, once against a healthy one — and compare. A difference is a finding; identical results mean you are looking at the wrong dimension.

# Degraded population: what do downstream calls cost?
{ span.http.route = "/checkout/{orderId}" && span:duration > 1.9s }
  >> { span:kind = client } | by(span.server.address) | avg(span:duration)

# Healthy population, same query
{ span.http.route = "/checkout/{orderId}" && span:duration < 400ms }
  >> { span:kind = client } | by(span.server.address) | avg(span:duration)
The same query against both populations isolates the cause Four downstream services compared between fast and slow checkout traces. The payments service averages 60 milliseconds in both. The customer service averages 30 milliseconds in both. The inventory service averages 45 milliseconds in fast traces and 1,240 in slow ones. The pricing service averages 20 milliseconds in both. Only inventory differs. Average downstream duration · fast traces vs slow traces fast slow payments 60 ms → 62 ms customers 30 ms → 31 ms inventory 45 → 1,240 ms pricing 20 ms → 21 ms Three dependencies are unchanged; one is 27× slower. That is the answer, and it took two queries.

Turning a finding into a permanent check

Once a query has explained an incident, it is worth keeping — with the healthy answer written next to it, so the next person can tell at a glance whether the number they are looking at is normal.

# Runbook entry: "Is the inventory dependency degraded?"
# Query:
{ span.http.route = "/checkout/{orderId}" }
  >> { span:kind = client && span.server.address = "inventory.internal" }
  | quantile_over_time(span:duration, .95)
# Healthy: under 120 ms. Above 400 ms, checkout p95 starts to move.

The “healthy” line is what makes this usable under pressure. A query that returns 380 ms means nothing on its own; the same number next to “healthy: under 120 ms” is a diagnosis.

Query cost and how to keep it low

Latency queries are among the more expensive shapes because a duration filter cannot be answered from an index — the engine must read spans. Three habits keep them fast: always lead with resource.service.name so whole blocks can be skipped; keep the time window as narrow as the question allows, starting at fifteen minutes around the event; and prototype the query on a narrow window before widening it. An aggregate over six hours can be orders of magnitude more expensive than the same aggregate over fifteen minutes, and during an incident that difference is the difference between an answer and a timeout.

It is also worth knowing when not to use a trace query at all. If the question is “when did p95 start rising”, a metric answers it instantly and a trace query answers it slowly. Traces are for explaining a change, metrics for detecting it — see generating RED metrics from spans for how to have both from the same data.

The shapes latency queries reveal

Once you have a set of traces from the degraded band, they tend to fall into a small number of shapes, and recognising which one you have determines the next query rather than the next guess.

A single dominant span. One child accounts for most of the parent’s duration. The cause is that operation, and the next question is why — a slow query, a saturated dependency, a lock. This is the easy case and the one people expect.

Uniform inflation. Every span is proportionally slower, with no single culprit. That pattern points at something affecting the whole process: CPU throttling, garbage collection, a noisy neighbour, or a node with degraded networking. Grouping the slow traces by host or pod usually confirms it in one query, and it is the shape most often misdiagnosed as an application problem.

Gaps between spans. The spans are fast, the total is slow, and the time is in the spaces between them. That is un-instrumented work — serialization, template rendering, business logic with no span — and the fix is instrumentation before it is optimisation. It is also the shape that makes people distrust tracing, because the trace appears to show that nothing took any time.

Many small spans, none significant alone. A run of short operations that add up. This is the N+1 family, and the diagnostic is count rather than duration, as covered in detecting N+1 queries from trace waterfalls.

Each shape has a different follow-up query, and misreading one for another sends the investigation in the wrong direction for an hour. The habit worth building is to classify the shape before forming a hypothesis about the cause.

Latency is a distribution, not a number

One habit underpins everything on this page: treat latency as a distribution at every step. A single trace is a sample from it, a percentile is a summary of it, and a change in latency is a change in its shape.

That framing prevents the two most common analytical errors. The first is generalising from one trace, which is sampling of size one. The second is comparing averages, which hides exactly the bimodal patterns — a fast path and a slow path, a cache hit and a cache miss — that most latency investigations are ultimately about. When the p50 is unchanged and the p95 has doubled, some population got slower and others did not, and finding which population is the entire investigation.

Query cost by shape, same 30-minute window fetch by trace id *milliseconds index lookup service + duration filter seconds block pruning works span attribute filter only *tens of seconds no pruning aggregate with by() minutes reads every match regex over attributes *minutes, unpredictable scan plus regex Query cost by shape, same 30-minute window query shape cost why fetch by trace id milliseconds index lookup service + duration filter seconds block pruning works span attribute filter only tens of seconds no pruning aggregate with by() minutes reads every match regex over attributes minutes, unpredictable scan plus regex Leading with a resource filter moves a query from the bottom rows to the top ones.

Common pitfalls

  • Sorting by slowest. Reliably returns the least representative traces in the window.
  • Using a round-number threshold. One second means nothing until you know whether this endpoint’s p95 is 80 ms or 900 ms.
  • Omitting the healthy comparison. Without it, every number looks suspicious and nothing is confirmed.
  • Confusing {A && B} with {A} && {B}. The first needs one span matching both conditions; the second needs two spans in the same trace. Silently different results.
  • Widening the window to find more results. If a narrow window returns nothing, the more likely explanation is sampling or a wrong attribute name, not an insufficiently large search.

Related

↑ Back to Querying Traces with TraceQL and Jaeger Search