Calculating Sampling Rates for a Trace Budget

Divide your monthly storage budget by (bytes per span × retention days × 86,400) to get a sustainable spans-per-second ceiling, then set the rate to that ceiling divided by your peak spans-per-second — and raise the rate on low-traffic routes until each keeps at least a few hundred traces per hour.

Context and when it matters

“We sample at 1%” is almost always an inherited number that nobody can defend. It was set when the fleet was six services, and since then the service count has tripled and the spans-per-request count has quadrupled, so the actual volume has grown twelvefold while the rate stayed put. Either the bill is now surprising or the coverage is now useless — usually one team is complaining about each.

A rate is a budget decision expressed as a probability. Doing the arithmetic explicitly takes fifteen minutes, produces a number you can defend in a cost review, and — more usefully — tells you which endpoints are being starved of coverage by a fleet-wide rate that was set for the busiest one.

The arithmetic

Four measured inputs and one decision produce the rate. Every input can be read from the pipeline you already run.

From budget to sample rate in four measurements Four input boxes feed into a calculation. Peak requests per second times spans per request gives unsampled spans per second. Bytes per span times retention days converts a storage budget into a spans-per-second ceiling. The ceiling divided by the unsampled rate gives the sample rate, shown here as 3.7 percent. Measure four things, decide one, divide peak requests/s 4,200 spans per request 63 bytes per span 420 (compressed) budget · retention 2 TB · 14 days unsampled span rate 264,600 spans/s affordable span rate 9,900 spans/s sample rate 3.7% 2 TB ÷ (420 bytes × 14 days × 86,400 s) ≈ 9,900 spans/s affordable · 9,900 ÷ 264,600 ≈ 3.7%

Measure, do not estimate

# Spans per request: total spans exported ÷ total requests, over the same window
curl -s http://otel-collector:8888/metrics | grep otelcol_receiver_accepted_spans
# 1,284,391 spans in 60s = 21,406 spans/s at a 1% sample
# → unsampled rate ≈ 2,140,600 spans/s? No — divide by the sample rate carefully:
#   sampled spans/s ÷ sample_rate = unsampled spans/s

# Bytes per span, as stored (compressed) rather than on the wire
#   Tempo:  block size ÷ spans in block
#   Jaeger: index stats ÷ document count

Compressed bytes per span typically lands between 300 and 600. Use your own number: a service that puts 2 KB SQL statements on every database span is nowhere near the same figure as one that does not.

Convert budget to ceiling, then to a rate

# sampling_budget.py — the whole calculation, with the assumptions visible
BUDGET_BYTES     = 2 * 1024**4      # 2 TB of trace storage at steady state
BYTES_PER_SPAN   = 420              # measured, compressed, as stored
RETENTION_DAYS   = 14
PEAK_RPS         = 4_200
SPANS_PER_REQ    = 63

seconds_retained = RETENTION_DAYS * 86_400
# Steady state: what you can store equals what you can ingest per second
# multiplied by how long you keep it.
affordable_spans_per_s = BUDGET_BYTES / (BYTES_PER_SPAN * seconds_retained)
unsampled_spans_per_s  = PEAK_RPS * SPANS_PER_REQ

rate = affordable_spans_per_s / unsampled_spans_per_s
print(f"affordable: {affordable_spans_per_s:,.0f} spans/s")   # 9,900
print(f"unsampled:  {unsampled_spans_per_s:,.0f} spans/s")    # 264,600
print(f"rate:       {rate:.1%}")                              # 3.7%

Two corrections to apply to that number before shipping it. Leave 30–40% headroom for traffic growth and incident-time retry storms, and remember that with head-based sampling the rate is decided per trace at the root, so a rate of 3.7% keeps 3.7% of traces complete, not 3.7% of spans scattered at random.

Allocate per service rather than fleet-wide

A single rate over-samples the busiest endpoint and starves everything else. Allocation fixes that without changing the total.

# Per-service rates via environment, keeping the same overall volume.
# checkout-api: 40% of traffic, gets 2% — still the largest contributor
# admin-api:    0.1% of traffic, gets 100% — costs almost nothing, and
#               without it an admin endpoint sees 4 traces a day
services:
  checkout-api:  { OTEL_TRACES_SAMPLER_ARG: "0.02" }
  search-api:    { OTEL_TRACES_SAMPLER_ARG: "0.05" }
  payments-api:  { OTEL_TRACES_SAMPLER_ARG: "0.25" }
  admin-api:     { OTEL_TRACES_SAMPLER_ARG: "1.0"  }

Give low-traffic routes a floor

Statistics, not fairness, sets the floor. To see roughly 1% of a route’s traffic in a way that reflects reality, you need enough retained traces that a single outlier is not the whole picture — a few hundred per hour is the usual working minimum.

Sample rate needed for usable coverage, by route volume Four rows. The search route at 2400 requests per second needs only 0.5 percent to retain about 43,000 traces per hour. The checkout route at 380 per second needs 2 percent. The refund route at 12 per second needs 25 percent. The admin export route at 0.2 per second needs 100 percent and still only retains 720 traces per hour. One rate cannot serve routes four orders of magnitude apart route req/s rate needed traces kept/h /search 2,400 0.5% 43,200 /checkout/{id} 380 2% 27,360 /refunds 12 25% 10,800 /admin/export 0.2 100% 720 Sampling the rare route at 100% adds 0.03% to total volume — the cheapest coverage you will ever buy.

What the rate does not buy you

A rate is a statement about volume, not about coverage of the things you care about. That distinction is worth being explicit about before anyone treats the calculated number as a guarantee.

At 3.7%, a route serving a thousand requests an hour keeps about thirty-seven traces. If a bug affects one request in five hundred, you will see roughly two examples an hour — enough to confirm the bug exists, not enough to characterise it. If it affects one in fifty thousand, you will see one every twenty-seven hours, and the first person to look will conclude the traces are broken rather than the sampling is honest. Rare failures are precisely the case head-based sampling handles worst, because the decision is made before anything is known about the outcome.

The practical mitigations are all forms of moving the decision or overriding it. A tail-based sampler buffers the trace and decides after seeing whether it failed, which converts “3.7% of everything” into “3.7% of successes and 100% of errors” at roughly the same storage cost. A forced-sample override — a header or a feature flag that sets the sampled flag for a specific user, tenant, or debug session — lets support capture a complete trace for a reported problem without touching the fleet rate. And a temporary rate increase on one route, applied while investigating, costs little because that route is a small fraction of total volume.

There is also a subtler limitation. Because the decision is made at the root and inherited, the rate you calculate governs whole traces, so any per-service analysis you do is over a biased sample: services that appear only in traces started by a particular gateway inherit that gateway’s rate, not their own. When comparing latency across services, check that both are represented in the same trace population before drawing conclusions — otherwise you are comparing a 10% sample of one against a 100% sample of another.

Verification

After changing rates, confirm the outcome rather than assuming it:

# Ingest rate against the calculated ceiling
sum(rate(otelcol_receiver_accepted_spans[5m]))          # should sit near 9,900

# Storage growth against the budget, projected to steady state
predict_linear(tempo_ingester_bytes_written_total[6h], 14 * 86400)

Then check coverage where it matters: query the retained trace count per route over an hour and confirm no important endpoint fell below the floor.

The shape you are looking for after a per-route reallocation is flat total volume with a much better distribution:

Same budget, redistributed across routes Two grouped bars per route. Under a flat 1 percent rate, search retains 86,000 traces per hour, checkout 13,000, refunds 432 and admin export 7. After per-route allocation, search retains 43,000, checkout 27,000, refunds 10,800 and admin export 720. Total volume is unchanged. Traces kept per hour · same total spend flat 1% allocated /search 86,400 → 43,200 /checkout/{id} 13,680 → 27,360 /refunds 432 → 10,800 /admin/export 7 → 720 The busiest route loses traces it did not need; every other route becomes debuggable. Nothing else changes.

Decision criteria

  • Is your storage cost dominated by trace volume? Then the rate is the lever. If it is dominated by retention, cut days before you cut rate — see sizing trace storage and retention costs.
  • Do you need every error trace? A probability rate cannot promise that. Use tail-based sampling for guaranteed retention of failures.
  • Are spans per request growing? Volume grows with them even at a fixed rate. Re-measure quarterly, and after any change to database or cache instrumentation.
  • Is the fleet heterogeneous? Then allocate per service; a single number is a compromise that suits no one.

Common pitfalls

  • Sampling below the parent. With parentbased_traceidratio, a downstream service with a lower rate than its caller does not reduce volume — the parent’s decision already won. Rate changes are only effective at the trace root.
  • Measuring bytes on the wire instead of at rest. OTLP compresses roughly 4:1; using uncompressed bytes overstates the requirement fourfold and produces an unnecessarily brutal rate.
  • Forgetting the ratio is per trace. 1% means one trace in a hundred kept in full, not one span in a hundred. The complete trace is the point.
  • Ignoring incident-time amplification. Retry storms multiply spans per request. Headroom is not optional; without it the pipeline sheds load exactly when the traces matter most.

Related

↑ Back to Choosing Between Head-Based and Tail-Based Sampling