Configuring the Batch and Memory Limiter Processors

In an OpenTelemetry Collector pipeline, memory_limiter must be the first processor and batch the last: the limiter rejects data before anything allocates memory for it, and batching only makes sense after every other processor has finished adding, dropping, or mutating spans.

Context & When It Matters

These two processors are the load-bearing walls of a Collector traces pipeline, and the failure modes they prevent are the ones that page you at 3 a.m. memory_limiter is what stands between a traffic spike and an OOMKilled Collector; batch is what keeps export throughput high enough that the Collector does not fall behind ingest. Misconfigure either and the symptoms are severe but misleading — spans vanish with no error in the SDK, the Collector restarts in a crash loop, or export latency climbs until the sending queue overflows. Getting these right is a prerequisite for everything else in the Collector pipeline configuration, because no amount of clever filtering or sampling survives a process that keeps getting killed.

Minimal Working Configuration

Start from the smallest configuration that behaves correctly, then tune the numbers to your workload.

processors:
  memory_limiter:
    check_interval: 1s      # how often heap usage is sampled
    limit_mib: 1500         # hard ceiling: refuse data above this
    spike_limit_mib: 500    # headroom for sudden bursts
  batch:
    timeout: 5s             # flush at least this often
    send_batch_size: 8192   # target spans per export
    send_batch_max_size: 10000  # hard cap per export request

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]   # limiter first, batch last
      exporters: [otlp]

The memory_limiter computes two thresholds. The soft limit is limit_mib - spike_limit_mib (here 1000 MiB); once heap crosses it, the limiter begins refusing new batches and returning errors to the receiver. The hard limit is limit_mib (1500 MiB); crossing it forces a garbage collection. check_interval sets how often these are evaluated — 1 second is responsive enough for most workloads without burning CPU on constant heap inspection.

batch accumulates spans until either send_batch_size is reached or timeout elapses, whichever comes first, then emits one grouped payload. send_batch_max_size caps the size of any single export so a sudden flood cannot produce a payload that exceeds the backend’s request size limit.

Implementation

The production configuration below adds sizing rationale as inline comments and pairs the limiter with the exporter’s own queue, which is where restart durability actually comes from.

processors:
  # FIRST in the chain: refuse data before any downstream processor
  # allocates memory copying and mutating it. If this ran later, the
  # processors ahead of it would already have done the allocation.
  memory_limiter:
    check_interval: 1s
    # Hard limit. Set to ~80% of the container memory limit so the
    # process has headroom for buffers the limiter does not track
    # (exporter queues, tail_sampling buffers, Go runtime overhead).
    limit_mib: 1500
    # Soft-limit headroom = limit_mib - spike_limit_mib = 1000 MiB.
    # Sized to absorb one check_interval of burst traffic without
    # tripping the hard limit. Must be smaller than limit_mib.
    spike_limit_mib: 500

  # LAST in the chain: group survivors into export-sized payloads only
  # after filter, transform, and sampling have finished changing them.
  batch:
    # Flush every 5s even if send_batch_size is not reached, so
    # low-traffic periods do not hold spans indefinitely.
    timeout: 5s
    # Target payload size. Larger amortizes export overhead but raises
    # per-batch memory and end-to-end latency. 8192 suits a busy gateway.
    send_batch_size: 8192
    # Absolute ceiling. Protects against a burst producing a payload
    # larger than the backend's max_recv_msg_size.
    send_batch_max_size: 10000

exporters:
  otlp:
    endpoint: tempo-distributor.monitoring.svc:4317
    # Durability against Collector restarts lives HERE, not in batch.
    sending_queue:
      enabled: true
      queue_size: 10000
    retry_on_failure:
      enabled: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      # Order is execution order. Nothing else about this line is negotiable.
      processors: [memory_limiter, batch]
      exporters: [otlp]

The sizing logic is a chain of dependencies. Container memory limit sets limit_mib at roughly 80 percent. limit_mib minus spike_limit_mib sets the soft limit, which should absorb one check_interval of peak ingest. send_batch_size times the number of concurrent batches must fit comfortably below the spike headroom, because a large batch is itself a memory allocation the limiter has to account for. This is the interplay that trips people up: a batch sized without reference to spike_limit_mib can single-handedly push the process over its hard limit. The two processors are tuned together, not independently. Because batching relies on holding spans in memory, it is also why tail_sampling — which buffers whole traces — has to be budgeted alongside them when all three share a pipeline.

A Worked Sizing Example

Numbers make the dependency chain concrete. Take a gateway Collector pod with a 2 GiB container memory limit ingesting 40,000 spans per second at peak, where each span averages roughly 1.5 KiB once attributes are attached.

Start from the container limit and work inward. Set limit_mib to 80 percent of 2048 MiB, which is 1638 MiB — round to 1600. That 20 percent gap is not slack you are wasting; it is reserved for allocations memory_limiter cannot see, namely the exporter’s sending_queue, the Go runtime’s own heap fragmentation, and any tail_sampling trace buffer sharing the pipeline. Next, size spike_limit_mib to cover one check_interval of burst. At 40,000 spans per second and 1.5 KiB each, one second of ingest is about 60 MiB of raw span data, and the copies made while processing inflate that several-fold. A spike_limit_mib of 400–500 MiB gives the limiter room to notice and react before the hard limit is breached. That leaves a soft limit of 1100–1200 MiB as the working ceiling.

Now size batch against what remains. A send_batch_size of 8192 spans at 1.5 KiB is roughly 12 MiB per in-flight batch. With a handful of batches queued behind a briefly slow exporter, that is tens of megabytes — comfortably inside the spike headroom. Push send_batch_size to 100,000 and a few queued batches alone could exceed spike_limit_mib and trip the hard limit, which is exactly how an innocent-looking batch tweak turns into a crash loop. The two knobs are coupled: every increase in batch size eats into the memory budget the limiter is trying to defend.

The table summarizes three tiers built from this reasoning.

Setting Low-traffic agent Busy gateway High-throughput gateway
Container memory limit 512 MiB 2 GiB 4 GiB
limit_mib 400 1600 3200
spike_limit_mib 100 500 800
check_interval 1s 1s 2s
send_batch_size 2048 8192 16384
timeout 5s 5s 2s

The high-throughput column shortens timeout to 2 seconds so a very high span rate flushes before batches grow past send_batch_max_size, and lengthens check_interval slightly because heap inspection itself costs CPU at that volume. These are starting points, not final values — the self-telemetry metrics are what tell you whether the guess held under real traffic.

Decision Criteria

Use this checklist to confirm the two processors are sized correctly for your deployment:

  • Is memory_limiter the first processor in every pipeline? If not, move it. A limiter that runs second protects nothing the first processor already allocated.
  • Is spike_limit_mib strictly less than limit_mib? The Collector refuses to start otherwise. Target 20–30 percent of the hard limit.
  • Is limit_mib about 80 percent of the container limit? The gap covers exporter queues and runtime overhead the limiter does not measure.
  • Is batch the last processor? Anything after it re-batches; anything sampled after it wastes the grouping.
  • Does send_batch_max_size sit below the backend’s request size limit? For Jaeger or Tempo, match it to the receiver’s max_recv_msg_size.
  • Is otelcol_processor_refused_spans flat at zero under normal load? A nonzero value means the soft limit is being hit routinely — raise limit_mib or scale out.

Verify the running configuration by scraping the Collector’s self-telemetry:

curl -s http://localhost:8888/metrics | grep -E \
  "otelcol_processor_refused_spans|otelcol_processor_batch_batch_send_size"

refused_spans climbing means the limiter is shedding load; a batch_send_size far below send_batch_size means traffic is too low to fill batches and timeout is doing the flushing.

Common Pitfalls

  • Placing batch before memory_limiter. The batch buffer allocates and holds up to send_batch_size spans before the limiter can reject anything, so a burst OOMs the process before protection engages. This is the single most damaging ordering mistake, and it produces crash loops rather than clean backpressure.
  • A too-aggressive memory_limiter causing silent drops. Setting limit_mib too low or spike_limit_mib too small trips the soft limit on ordinary bursts. The limiter refuses data, backpressure reaches the sending SDK, and if its queue is full the SDK drops spans — invisibly, because the loss happens upstream of the Collector’s own metrics.
  • Tiny batches killing throughput. A very small send_batch_size or a sub-second timeout floods the exporter with high-frequency requests, saturating its connection pool and inflating per-span export cost. Fewer, fuller batches almost always beat many small ones for a busy pipeline.

FAQ

What happens if I set spike_limit_mib higher than limit_mib?

The Collector fails to start. spike_limit_mib is the headroom subtracted from limit_mib to compute the soft limit, so it must be smaller than limit_mib. A common working ratio is a spike limit of roughly 20 to 30 percent of the hard limit.

Does the batch processor guarantee spans are never lost?

No. batch holds spans in an in-memory buffer, so an OOM kill or an unclean shutdown loses whatever is buffered. It also has no retry logic of its own. Durability against restarts comes from the sending_queue and a persistent storage extension on the exporter, not from batch.

Why did lowering the batch timeout increase my export error rate?

A very short timeout flushes tiny batches at high frequency, multiplying the number of export requests. That can saturate the exporter’s connection pool or the backend’s ingest limits, surfacing as export errors. Raise the timeout or send_batch_size so each request carries more spans.

↑ Back to OpenTelemetry Collector Pipeline Configuration