OpenTelemetry Collector Pipeline Configuration

Problem Framing

A team ships instrumentation, points every service at a single Collector, and within a week the Collector pod is OOMKilled twice an hour. Traces arrive in Jaeger with duplicated attributes, health-check spans drown out real requests, and a sudden traffic spike causes the Collector to silently discard a third of all spans. Each of these is a pipeline-configuration failure, not an SDK bug. The Collector is a programmable stream processor: raw spans enter through receivers, pass through an ordered chain of processors that limit, enrich, filter, reshape, and sample them, and leave through exporters. Get the order or the memory budget wrong and the pipeline either loses data or corrupts it. This guide builds a complete traces pipeline for the OpenTelemetry Collector, explains why memory_limiter must come first and batch must come last, contrasts agent and gateway deployment, and shows how the spanmetrics connector turns your span stream into RED metrics.

Prerequisites

Anatomy of a Traces Pipeline

A Collector pipeline is three ordered stages plus an optional connector that bridges signal types. Receivers accept data in a wire format (OTLP over gRPC or HTTP, Zipkin, Jaeger legacy) and normalize it into the internal pdata representation. Processors form a strict linear chain — each one receives the batch the previous one emitted, so their order is the order of mutation. Exporters serialize the processed spans and ship them to a backend. Connectors are the newer primitive: a component that is an exporter on one pipeline and a receiver on another, letting you derive metrics from traces without an external process.

The single most important rule is that processor order is defined by the processors: array inside each pipeline, not by the order in which processors are declared. The Collector reads that array top to bottom and pipes spans through it in exactly that sequence. The diagram below shows the canonical order and why each position is fixed.

Collector Traces Pipeline Order Spans enter through the OTLP receiver, pass through processors in the fixed order memory_limiter, resource, attributes, filter, transform, tail_sampling, batch, then leave through the OTLP exporter to a backend. A spanmetrics connector taps the stream and feeds a separate metrics pipeline. order in the processors: array is the order of execution otlp receiver processor chain memory_limiter1st resourceattrs filter transform tail_sampling batchlast otlp Jaeger / Tempo spanmetrics connector → metrics

Each stage has a job. memory_limiter is the circuit breaker that protects the process from OOM. resource and attributes enrich spans with deployment identity and normalized keys. filter discards spans you never want stored. transform rewrites the spans you keep — redacting, renaming, hashing. tail_sampling decides which complete traces survive. batch groups the survivors into efficient export payloads. The exporter ships them.

Agent Versus Gateway Deployment

There are two deployment topologies, and most production systems run both. An agent Collector runs close to the workload — as a DaemonSet on every node or a sidecar in every pod. It does cheap, local work: receiving OTLP over localhost, adding host metadata, and forwarding quickly. A gateway Collector runs as a horizontally scaled standalone deployment that all agents forward to. It does the expensive, stateful work that needs a global view: tail_sampling, spanmetrics aggregation, and fan-out to storage backends.

The division matters most for tail_sampling. That processor must see every span of a trace to decide whether to keep it, so all spans sharing a trace ID must land on the same Collector instance. Agents see only the spans produced on their node, so a sampling decision there operates on a fragment. The fix is a two-tier design: agents forward with a loadbalancing exporter that hashes on trace ID, guaranteeing that a full trace converges on one gateway replica, and the gateway tier owns sampling. For context propagation across service meshes, the agent tier is also where sidecar-injected headers get reconciled before spans leave the node.

Step-by-Step Implementation

The following otel-collector-config.yaml is a complete gateway configuration. Each subsequent step explains one region of it.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        max_recv_msg_size_mib: 16
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1500
    spike_limit_mib: 500
  resource:
    attributes:
      - key: collector.tier
        value: gateway
        action: upsert
  attributes:
    actions:
      - key: http.request.header.authorization
        action: delete
      - key: db.statement
        action: hash
  filter/health:
    error_mode: ignore
    traces:
      span:
        - 'attributes["http.route"] == "/healthz"'
        - 'attributes["http.route"] == "/readyz"'
  transform/rename:
    error_mode: ignore
    trace_statements:
      - context: span
        statements:
          - set(name, Concat([attributes["http.method"], attributes["http.route"]], " ")) where attributes["http.route"] != nil
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 750 }
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }
  batch:
    timeout: 5s
    send_batch_size: 8192
    send_batch_max_size: 10000

connectors:
  spanmetrics:
    histogram:
      explicit:
        buckets: [10ms, 50ms, 100ms, 250ms, 1s, 5s]
    dimensions:
      - name: http.route
      - name: service.name

exporters:
  otlp/tempo:
    endpoint: tempo-distributor.monitoring.svc:4317
    tls:
      insecure: false
  prometheus:
    endpoint: 0.0.0.0:8889
  debug:
    verbosity: normal

service:
  extensions: [zpages]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resource, attributes, filter/health, transform/rename, tail_sampling, batch]
      exporters: [otlp/tempo, spanmetrics]
    metrics/spanmetrics:
      receivers: [spanmetrics]
      processors: [memory_limiter, batch]
      exporters: [prometheus]

extensions:
  zpages:
    endpoint: 0.0.0.0:55679

Step 1 — Define the OTLP receiver

The otlp receiver exposes gRPC on 4317 and HTTP on 4318. gRPC is the default for backend-to-Collector traffic because it multiplexes and uses binary framing; HTTP is useful for browser SDKs and environments where a proxy mangles gRPC. Setting max_recv_msg_size_mib guards against a single oversized batch — common when a service attaches large payloads as span attributes — exhausting memory before memory_limiter even sees it.

Step 2 — Place memory_limiter first

memory_limiter periodically checks the process’s heap. When usage crosses limit_mib, it begins refusing new data and returns an error to the receiver; when it crosses limit_mib + spike_limit_mib, it forces garbage collection. Because its entire job is to reject data before downstream processors allocate memory copying and mutating it, it must be the first entry in the processors array. Placing it anywhere else means the processors ahead of it have already done the allocation you were trying to prevent. Tuning specifics live in configuring the batch and memory limiter processors.

Step 3 — Enrich with resource and attributes

resource operates on the resource-level attributes shared by every span in a batch — service.name, deployment.environment, and here a collector.tier marker. attributes operates on individual span attributes; in this config it deletes the authorization header captured by an over-eager instrumentation and hashes db.statement so query text is correlatable but not readable. Doing enrichment before sampling means the sampling policies and derived metrics see the final, normalized keys.

Step 4 — Drop noise with filter, reshape with transform

filter/health removes liveness and readiness probe spans using OTTL boolean conditions — these fire thousands of times an hour and carry no diagnostic value. transform/rename uses OTTL statements to set a low-cardinality span name from the HTTP method and route, so a downstream metrics view groups cleanly. Filtering before transforming avoids wasting rewrite work on spans that are about to be discarded. The full OTTL vocabulary for both is covered in filtering and transforming spans in the Collector.

Step 5 — Apply tail_sampling

tail_sampling buffers spans for decision_wait seconds so it can assemble complete traces, then applies its policies in order: keep anything with an error status, keep anything slower than 750 ms, and otherwise keep 5 percent probabilistically. A trace matching any policy is retained in full. This is the tail-based sampling decision that head sampling at the SDK cannot make, because it needs the finished trace to know whether it erred or was slow.

Step 6 — Batch last, then export

batch groups spans into export-sized payloads. It is last because everything upstream may add, drop, or mutate spans; batching earlier would group spans that a later processor then modifies, forcing re-batching and wasting the grouping. After batching, the otlp/tempo exporter ships survivors to Tempo over TLS.

Step 7 — Derive RED metrics with spanmetrics

The spanmetrics connector sits in the traces pipeline’s exporters list and in a metrics pipeline’s receivers list. Every span that reaches it increments request-count and error-count counters and records a latency histogram, dimensioned by route and service. Those series flow into the metrics pipeline and out to Prometheus, feeding trace-based alerting and SLO monitoring without a separate metrics SDK. Note the placement: because spanmetrics receives the traces pipeline’s output, it only counts spans that survived tail_sampling — if you want unsampled RED metrics, tap the stream on a pipeline branch before the sampler.

Step 8 — Wire and verify the service block

The service.pipelines block is where the components become a running graph. Nothing you declare in receivers, processors, or exporters executes until it is referenced by a pipeline. The traces pipeline lists its processors in the mandatory order; the metrics/spanmetrics pipeline consumes the connector’s output. zpages is registered as an extension for live introspection.

Verification

Confirm the pipeline is healthy with three independent checks.

First, add the debug exporter to the traces pipeline temporarily and read stdout to see post-processing spans:

kubectl logs -l app=otel-collector | grep -A3 "span.name"

Second, use zPages to inspect live pipeline throughput and per-component span counts:

kubectl port-forward svc/otel-collector 55679:55679
# then open the pipeline view
curl -s "http://localhost:55679/debug/pipelinez" | grep -i traces

Third, scrape the Collector’s own telemetry to confirm spans are accepted, not refused or dropped:

curl -s http://localhost:8888/metrics | grep -E \
  "otelcol_receiver_accepted_spans|otelcol_processor_refused_spans|otelcol_exporter_sent_spans"

A healthy gateway shows accepted_spans climbing, sent_spans tracking it after the sampling ratio is applied, and refused_spans flat at zero. A rising refused_spans total means memory_limiter is shedding load — the signal to scale out or raise the limit.

Edge Cases & Gotchas

  1. memory_limiter placed after batch. If batch runs first, it allocates and holds up to send_batch_size spans in memory before memory_limiter can reject anything. Under a burst, the process OOMs before the limiter engages. The limiter must precede every allocating processor.

  2. batch before tail_sampling. Batching first groups spans that the sampler then splits by trace-level decisions, breaking the batch and negating the efficiency. Worse, some processors expect to see spans grouped by trace, which batching can scramble. Keep batch strictly last.

  3. Backpressure silently dropping data. When memory_limiter refuses data, the receiver returns an error to the sender. If the upstream agent or SDK has a bounded, already-full retry queue, it drops the overflow. The loss is invisible in the gateway’s metrics — it shows as refused_spans on the gateway but as dropped_spans on the sender. Monitor both sides.

  4. tail_sampling on an agent. Running tail_sampling per node evaluates partial traces and produces inconsistent keep decisions across a single trace, yielding fragmented traces in the backend. Move it to a gateway fed by a trace-ID-hashing load balancer.

  5. filter deleting a parent span. Dropping a mid-trace span (not just a leaf health check) orphans its children, which then reference a missing parent. Backends render these as broken trees. Filter only spans that are safe to remove wholesale — see the span lifecycle rules for what “safe” means.

  6. Connector referenced in only one pipeline. spanmetrics produces nothing unless it appears as an exporter on the traces pipeline and a receiver on a metrics pipeline. Referencing it on only one side is a silent no-op.

Performance & Scale Notes

Sizing the gateway. tail_sampling holds num_traces traces in memory for decision_wait. At 50,000 traces and a 10-second window, plan for well over a gigabyte of resident buffer under load; this is the dominant memory consumer and the reason limit_mib and num_traces must be tuned together.

Horizontal scale requires trace affinity. You cannot simply add gateway replicas behind a round-robin service if you run tail_sampling — spans of one trace would scatter across replicas. The loadbalancing exporter on the agent tier, keyed on trace ID, is mandatory for a multi-replica sampling gateway.

Batch size versus latency. Larger send_batch_size amortizes export overhead but increases end-to-end latency and per-batch memory. The interplay between batch and memory_limiter — where a large batch can push the process over its spike limit — is the single most common tuning mistake, examined in depth in the batch and memory limiter guide.

Processor cost order. OTTL regex in transform and multi-policy tail_sampling are the most expensive stages. Filtering cheap-to-match noise early (health checks by exact route string) shrinks the volume those expensive stages must process, so ordering cheap drops before expensive rewrites is both a correctness and a throughput decision.

Troubleshooting FAQ

Does the order of processors in the YAML list actually matter?

Yes. The Collector runs processors in the exact order they are listed in the pipeline’s processors array, not the order they are defined in the processors block. memory_limiter must be first so it can reject data before downstream processors allocate memory, and batch must be last so grouping happens after all per-span mutation and sampling decisions.

Why is the Collector dropping spans under load even though the SDK is not?

When memory_limiter crosses its hard limit it refuses new data and returns an error to the receiver, which propagates backpressure to exporters. If the sending SDK’s queue cannot absorb the retry it drops spans. Raise limit_mib, add Collector replicas, or reduce upstream volume with head-based sampling before it reaches the gateway.

Should tail_sampling run in the agent or the gateway Collector?

Always the gateway. tail_sampling requires all spans of a trace to reach the same Collector instance so it can evaluate the complete trace. Agents run per-node and only see a fraction of any trace, so a tail_sampling policy there would make decisions on partial traces. Route by trace ID with a load-balancing exporter into a gateway tier that owns sampling.

How do I see what a span looks like after the processor chain runs?

Add a debug exporter with verbosity set to detailed to the same pipeline and read the Collector’s stdout. It prints every resource attribute, span name, and attribute after all processors have mutated the span, which lets you confirm that filter, transform, and attributes changes took effect before export.

Why are my spanmetrics missing or showing zero request counts?

The spanmetrics connector only counts spans from pipelines that list it as an exporter, and only emits into pipelines that list it as a receiver. If the traces pipeline does not export to spanmetrics, or the metrics pipeline does not receive from it, no series are produced. Also confirm the connector runs before tail_sampling drops the spans you want counted.

↑ Back to SDK Implementation & Context Propagation