Handling Collector Backpressure and Queue Overflow

Backpressure has exactly four observable stages — receiver refused, processor dropped, exporter queue full, and export failed — and each has its own counter, so the first step is always to read the counters rather than to increase a limit and hope.

Context and when it matters

The Collector never blocks your application. When it cannot keep up it sheds telemetry, quietly, and the only evidence is a counter nobody is watching. That design is correct — telemetry must never take down the workload it observes — but it means capacity problems present as “some traces are missing sometimes”, which is the least diagnosable symptom in the system.

Backpressure appears in three situations: a genuine traffic spike, a backend outage where the exporter cannot drain, and an incident where retry storms multiply span volume at exactly the moment you need traces most. The third is the one that matters, because it is self-reinforcing: more errors produce more spans, which fill the queue, which drops spans, which removes the evidence.

The four stages

Where a Collector sheds load, and the counter for each Spans enter the receiver, pass through the memory limiter and processors, enter the exporter sending queue, and are exported. Four shed points are marked: receiver refused spans under memory pressure, processor dropped spans, exporter enqueue failures when the queue is full, and export send failed spans when the backend rejects or times out. Four stages, four counters — read them before changing anything receiver accepts OTLP processors limit · filter · batch sending queue buffers for retry exporter writes to backend refused_spans memory limiter is shedding dropped_spans a filter matched (often on purpose) enqueue_failed queue is full — backend too slow send_failed_spans backend rejected or timed out Refusals mean you are CPU or memory bound. Enqueue failures mean the backend is the bottleneck. The fixes are different. Raising the queue when the receiver is refusing makes things worse — it consumes the memory the limiter was protecting. Counter to bottleneck, in one table receiver_refused_spans memory or CPU bound *scale out or reduce volume processor_dropped_spans a filter matched usually intentional exporter_enqueue_failed backend too slow *queue size or consumers exporter_send_failed backend errors check health and TLS queue_size ≈ capacity about to fail act before it saturates Counter to bottleneck, in one table counter bottleneck action receiver_refused_spans memory or CPU bound scale out or reduce volume processor_dropped_spans a filter matched usually intentional exporter_enqueue_failed backend too slow queue size or consumers exporter_send_failed backend errors check health and TLS queue_size ≈ capacity about to fail act before it saturates Reading the counters first turns a guessing exercise into a two-minute diagnosis.

Diagnosis

curl -s http://otel-collector:8888/metrics | grep -E \
  'receiver_refused_spans|processor_dropped_spans|exporter_enqueue_failed_spans|exporter_send_failed_spans|exporter_queue_size|exporter_queue_capacity|process_runtime_total_alloc'

Read the result as a decision table:

What is non-zero Bottleneck Fix
receiver_refused_spans Collector memory or CPU Scale out, raise limit_mib if headroom exists, reduce span volume
processor_dropped_spans A filter is matching Usually intentional — confirm the filter’s scope
exporter_enqueue_failed_spans Backend cannot keep up Increase queue_size, add exporter concurrency, scale the backend
exporter_send_failed_spans Backend errors or timeouts Check backend health, TLS, and the retry configuration
queue_size near queue_capacity Approaching the previous case Act before it saturates

Sizing the queue

The sending queue exists to survive a backend outage without losing data. Its size should therefore come from the outage duration you want to absorb, not from a round number:

# queue_sizing.py
SPANS_PER_SEC   = 20_000      # steady state through this collector
OUTAGE_SECONDS  = 120         # backend restart you want to ride out
BATCH_SIZE      = 8_192       # send_batch_size

# The queue counts BATCHES, not spans — the single most common sizing error.
batches_per_sec = SPANS_PER_SEC / BATCH_SIZE          # ≈ 2.4
queue_size      = int(batches_per_sec * OUTAGE_SECONDS * 1.3)   # ≈ 380

# Memory cost of that queue: batches × spans per batch × bytes per span
mem_mb = queue_size * BATCH_SIZE * 800 / 1024**2       # ≈ 2,400 MB
print(queue_size, f"{mem_mb:.0f} MB")

That last line is the point: a queue large enough to survive a two-minute outage at 20,000 spans/second needs roughly 2.4 GB of memory. If the container has 2 GB, the queue you configured cannot exist, and the memory limiter will start refusing spans long before the queue fills.

processors:
  memory_limiter:
    check_interval: 1s
    # Set BELOW the container limit — the limiter needs room to react.
    # Container 8Gi → limit_mib 6000, spike_limit_mib 1000.
    limit_mib: 6000
    spike_limit_mib: 1000

exporters:
  otlp/tempo:
    endpoint: tempo-distributor:4317
    sending_queue:
      enabled: true
      queue_size: 380
      num_consumers: 20          # parallel senders; raise if the backend allows
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      # Beyond this the data is stale anyway — stop holding memory for it.
      max_elapsed_time: 300s

num_consumers is underused. A single consumer serialises export, so a backend with 40 ms latency caps throughput at 25 batches/second regardless of queue size. Raising it to 20 raises the ceiling twentyfold, provided the backend can take the concurrency.

Choosing what to shed

When capacity genuinely cannot be increased in time, shedding deliberately beats shedding randomly. The Collector drops whatever arrives when full, which is uniform across value — the error trace you need is as likely to go as a health check.

# Shed the cheap traffic first, so pressure never reaches valuable spans.
processors:
  filter/drop_noise:
    error_mode: ignore
    traces:
      span:
        # Health checks and metrics scrapes: no diagnostic value, high volume.
        - 'attributes["http.route"] == "/healthz"'
        - 'attributes["http.route"] == "/metrics"'
        # Successful cache hits — the miss path is what explains latency.
        - 'attributes["db.system.name"] == "redis" and attributes["cache.hit"] == true'
service:
  pipelines:
    traces:
      # Filter BEFORE the batch processor so dropped spans never occupy a batch.
      processors: [memory_limiter, filter/drop_noise, batch]

Ordering matters: memory_limiter first so it can refuse before work is done, the filter next so noise is discarded cheaply, and batch last so batches contain only spans that will actually be exported.

Verification

# Any of these being non-zero is silent data loss — alert on all four
sum(rate(otelcol_receiver_refused_spans[5m])) > 0
sum(rate(otelcol_exporter_enqueue_failed_spans[5m])) > 0
sum(rate(otelcol_exporter_send_failed_spans[5m])) > 0

# Queue utilisation — page before it saturates, not after
max(otelcol_exporter_queue_size / otelcol_exporter_queue_capacity) > 0.7

Then test the failure deliberately: stop the backend for sixty seconds and confirm the queue absorbs the outage and drains afterwards with no enqueue_failed increments. A queue that cannot survive a planned restart will not survive an unplanned one.

Planning capacity instead of reacting to it

Backpressure is a capacity problem that announces itself late. Three numbers, tracked continuously, turn it into a planned one.

Headroom against peak. Compare sustained ingest against the rate at which refusals begin. Finding that threshold requires a load test rather than an outage: push a staging Collector until receiver_refused_spans becomes non-zero, and record the spans-per-second at which it happened. A production tier running at more than sixty percent of that figure has no room for an incident, and incidents are precisely when span volume multiplies.

Spans per request, tracked over time. Volume grows for two reasons — more traffic and more spans per request — and only the first is visible on a traffic dashboard. A team adding database and cache instrumentation can triple pipeline load without a single extra user, which is why the ratio deserves its own graph.

Time to drain. After a backend outage, how long does the queue take to empty? If draining takes longer than the outage lasted, the exporter is the bottleneck and no queue size will save you — the fix is more consumers or a faster backend. This is measurable deliberately: stop the backend for sixty seconds in a staging environment and watch the recovery curve.

With those three numbers, capacity conversations become concrete. Without them, the usual pattern is to raise a limit after every incident, arriving eventually at a configuration whose values nobody can justify and which still fails under a load nobody has measured.

What to shed, in order of preference What to shed, in order of preference. health checks: no value; cache hits: low value; baseline successes: some value; errors: never What to shed, in order of preference health checks no value cache hits low value baseline successes some value errors never Shedding deliberately keeps the traces that explain failures; shedding by overflow keeps whatever arrived first.

Common pitfalls

  • Raising queue_size when the receiver is refusing. The queue consumes the memory the limiter was defending; the refusal rate goes up, not down.
  • Setting limit_mib equal to the container limit. The limiter needs headroom to act before the kernel OOM-kills the process.
  • Leaving num_consumers at 1. Export throughput is then bounded by backend latency regardless of any other tuning.
  • Treating processor_dropped_spans as a fault. It usually means a filter is doing its job; confirm the filter’s scope before investigating capacity.
  • No alerts on the counters. Every one of these is silent by default, which is why missing traces are typically noticed days later by a human.

One structural mitigation is worth mentioning: a persistent queue backed by disk rather than memory survives a Collector restart, so a rolling upgrade during a backend outage does not discard everything buffered. It costs disk I/O and adds a failure mode of its own, but for pipelines where losing an hour of telemetry is genuinely expensive, it is the only option that survives a process restart.

Whatever else you change, keep the four counters on a dashboard that someone actually looks at. Every fix on this page is straightforward once you know which stage is shedding, and almost impossible to reason about when you do not.

Related

↑ Back to OpenTelemetry Collector Pipeline Configuration