Sizing Trace Storage and Retention Costs

Storage cost is spans/second × bytes per span × index multiplier × retention seconds, and of those four terms only two are worth optimising — bytes per span (attribute discipline) and retention (tiering) — because the other two are set by your traffic and your sampling policy.

Context and when it matters

Trace storage bills surprise people because three of the four terms grow independently. Traffic grows with the business. Spans per request grow every time someone instruments a new library — adding database and cache instrumentation routinely triples span volume overnight. Attribute size grows as teams add context. Nobody changed the sampling rate, and the bill doubled.

Modelling this before you commit to a backend, a retention window, or a vendor contract takes an afternoon and changes the decision surprisingly often. It also tells you which lever to pull when the bill does move: dropping retention from 30 days to 14 halves the cost with almost no impact on debugging, while halving the sample rate halves your chance of having the trace you need.

What a span actually costs at rest

Where the bytes in a stored span go Two stacked bars. A lean span totals about 180 bytes compressed: 48 bytes of identifiers, 24 of timing, 30 for the name, 60 for eight attributes and 18 of index overhead. A heavy span totals about 1400 bytes: the same identifiers and timing, plus 900 bytes of attributes dominated by a SQL statement, 300 bytes of exception stack trace and 130 of index overhead. Attributes are the only term you control lean span · 180 bytes ids 48 · timing 24 · name 30 · attrs 60 · index 18 heavy span · 1,400 bytes attrs 900 (SQL statement dominates) stack 300 index Identifiers and timing are fixed at ~72 bytes per span. Everything above that is a decision someone made. Truncating SQL statements to 512 characters typically cuts a data-heavy fleet's storage by a third.

Index overhead is where the two common backends diverge sharply. Tempo indexes only trace IDs plus a small set of block-level summaries, so overhead is roughly 5–15% and the object store holds compressed blocks. Jaeger with Elasticsearch indexes every tag by default, and the inverted index frequently exceeds the size of the documents themselves — a 2–3× multiplier is normal on attribute-rich spans. That difference dominates the comparison more than raw storage pricing does.

Do the arithmetic

# storage_model.py — everything in one place, so assumptions are visible
SPANS_PER_SEC     = 9_900          # post-sampling, measured at the collector
BYTES_PER_SPAN    = 420            # compressed, at rest, measured
INDEX_MULTIPLIER  = 1.10           # Tempo ~1.05-1.15, ES-backed Jaeger ~2-3
RETENTION_DAYS    = 14

daily_bytes = SPANS_PER_SEC * BYTES_PER_SPAN * INDEX_MULTIPLIER * 86_400
steady_state = daily_bytes * RETENTION_DAYS

print(f"per day:      {daily_bytes / 1024**4:.2f} TB")     # 0.35 TB
print(f"steady state: {steady_state / 1024**4:.2f} TB")    # 4.9 TB

# Object storage at ~$0.023/GB-month:
gb = steady_state / 1024**3
print(f"storage:      ${gb * 0.023:,.0f}/month")           # ~$115/month

# The line people forget: request costs. Tempo issues GET requests per query,
# and a busy dashboard can rival storage cost on its own.

Two figures from that model are worth internalising. First, raw object storage for a mid-sized fleet is cheap — hundreds of dollars, not thousands. Second, the expensive parts are elsewhere: the compute that ingests and queries, the index if you chose a backend that builds one, and the egress if your queries cross a region boundary. A cost model that stops at gigabytes-times-price understates the real number by a large factor.

Retention tiers beat a single window

Debugging value decays fast. Most trace lookups happen within an hour of the event, nearly all within a day, and what remains after that is compliance and capacity planning rather than debugging.

How quickly stored traces stop being queried A declining bar chart of query volume against trace age. Traces under one hour old account for the majority of queries. Volume falls sharply through the first day and is negligible after seven days. Two tier boundaries are marked: a hot tier covering the first three days and a cold tier covering the remainder. Queries by trace age · one month of usage <1h <6h 1d 2d 3d 5d 7d 10d 14d 21d 30d hot tier · 3 days · fast queries cold tier · cheap storage, slower reads Over 90% of queries land in the hot tier, which is under a quarter of the stored bytes. 
# tempo.yaml — a hot local tier with a longer object-store tail
storage:
  trace:
    backend: s3
    s3:
      bucket: traces-prod
    blocklist_poll: 5m
    # Local cache serves the first few days from disk; older blocks come
    # from the object store on demand.
    cache: memcached
    block:
      version: vParquet4

compactor:
  compaction:
    # This is the retention lever. Halving it halves steady-state storage.
    block_retention: 336h        # 14 days
    compacted_block_retention: 1h

The costs that are not storage

Object storage is usually the smallest line in a tracing bill, which is why models that only count gigabytes mislead. Four other costs matter, and their relative sizes differ enough between deployments that it is worth checking which one dominates yours before optimising anything.

Ingest compute. Collectors and ingesters must handle peak throughput plus incident-time amplification, and they are sized for that peak rather than for the average. A pipeline handling 10,000 spans/second at steady state typically runs several collector replicas and a similar number of ingesters, and that compute frequently costs more per month than the storage it feeds.

Query compute. Tempo’s queriers decompress blocks on demand, so an expensive aggregate over a wide window is real CPU. Dashboards that run TraceQL aggregates on a thirty-second refresh can dominate query cost while nobody is watching them.

Object-store request charges. Fetching a trace is several GET requests; a busy environment issuing millions of them per month pays a request bill that can rival the storage bill. Caching the hot tier locally is as much a cost control as a latency one.

Egress. Traces crossing a region or cloud boundary — a collector in one region exporting to a backend in another, or a vendor endpoint outside your network — are billed per gigabyte at rates that dwarf storage pricing. This is the single most common cause of a tracing bill that is an order of magnitude above the model.

The practical consequence is that the cheapest architecture is usually the one that keeps the pipeline close to the workload: collect in-region, sample before the expensive hop, and only move what you intend to keep.

Verification

# Actual bytes written per second, against the model
rate(tempo_ingester_bytes_written_total[1h])

# Bytes per span, measured rather than assumed
rate(tempo_ingester_bytes_written_total[1h])
  / rate(tempo_ingester_spans_received_total[1h])

# For Jaeger on Elasticsearch, the index multiplier that actually applies
elasticsearch_indices_store_size_bytes / elasticsearch_indices_docs_primary

Compare the measured bytes-per-span against your model quarterly and after any instrumentation change. A jump usually traces back to one team adding an attribute — a payload body, a full URL, a stack trace on a frequent error — rather than to traffic growth.

Decision criteria

  • Cost dominated by retention? Tier it. Three days hot plus eleven cold typically costs 40% of a flat fourteen and changes nothing about day-to-day debugging.
  • Cost dominated by index? That is a backend decision, not a tuning one. An attribute-rich fleet on Elasticsearch-backed Jaeger pays a 2–3× multiplier that Tempo simply does not charge — see Jaeger vs Tempo vs Zipkin.
  • Cost dominated by span volume? Look at spans per request before touching the sample rate. Aggregating forty cache spans into one with a count attribute is free coverage; halving the sample rate is not.
  • Compliance requires long retention? Retain a filtered subset — error traces and audited endpoints only — rather than everything. See configuring Jaeger retention policies for compliance.

The three levers, ranked by what they cost you in debugging ability per unit of saving:

Three ways to halve the bill, ranked by what they cost you Three rows. Trimming attributes such as SQL statements and stack traces saves about 35 percent and costs nothing in debugging ability. Tiering retention into hot and cold saves about 60 percent and costs only slower queries on old traces. Halving the sample rate saves 50 percent and costs half of all traces, including the one you will need. Pull them in this order 1 · trim attributes −35% costs nothing you were using 2 · tier retention −60% old traces read slower 3 · halve the sample rate −50% half your traces are gone Levers 1 and 2 together usually beat lever 3 outright — and they leave the coverage you paid for intact.

Common pitfalls

  • Modelling with uncompressed bytes. OTLP compresses roughly 4:1 and stored blocks compress further. Using wire bytes overstates storage several-fold and leads to an unnecessarily aggressive sample rate.
  • Forgetting request and egress charges. Object-store GET pricing and cross-region egress can exceed storage on a query-heavy deployment.
  • Assuming the multiplier is constant. Index overhead scales with attribute cardinality, not span count, so adding one high-cardinality attribute can move it materially.
  • Modelling steady state on day one. Storage grows for a full retention period before it plateaus, so the first two weeks look reassuringly cheap and the invoice at the end of the month is double what anyone expected. Always quote the steady-state figure, and label the ramp explicitly when reporting early numbers.
  • Sizing for average rather than peak. Ingest capacity must handle peak plus incident-time amplification; storage can be sized for the average, but the two are different calculations and conflating them under-provisions the ingest path.

Related

↑ Back to Trace Storage Backend Comparison: Jaeger vs Tempo