Deploying the Collector as Agent vs Gateway

Run an agent on every node for local buffering, host metadata, and a short export path; run a gateway tier when you need anything that requires seeing more than one node’s traffic — tail sampling, fleet-wide redaction, per-tenant routing, or a single egress point.

Context and when it matters

The Collector is one binary with two very different jobs, and most confusion about it comes from treating those jobs as interchangeable. As an agent it is a local sidecar or DaemonSet: it receives spans over localhost, enriches them with host and container metadata, batches, and forwards. As a gateway it is a horizontally scaled service: it receives from many agents, applies processing that needs a global view, and exports to storage.

Choosing wrongly has a specific cost. Agents only, and you cannot tail sample or apply a rate limit that means anything fleet-wide. Gateway only, and every service pays a network hop for every batch, a gateway restart drops in-flight spans across the whole estate, and you lose node-level enrichment entirely.

What each tier can see

Visibility determines which processors can work On the left, three nodes each run an agent that sees only its own pods' spans, permitting batching, host enrichment and local retry. On the right, a gateway tier receives from all agents and sees the whole fleet, permitting tail sampling, fleet-wide rate limiting, cross-service redaction and per-tenant routing. A processor can only act on what its tier receives node 1 · agent sees: pods on node 1 localhost:4317 node 2 · agent sees: pods on node 2 localhost:4317 node 3 · agent sees: pods on node 3 localhost:4317 gateway tier sees: the whole fleet scaled independently agent can do batch · host metadata local retry · k8sattributes memory limit per node only the gateway can do tail sampling fleet-wide rate limits per-tenant routing single egress + credentials Most production estates run both: the agent for locality and enrichment, the gateway for decisions that need the whole picture.

Configuration for each tier

# agent.yaml — DaemonSet, one per node. Small, fast, enriching.
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 400              # agents share the node with your workloads
    spike_limit_mib: 100
  # Node-local metadata that only the agent can know cheaply.
  k8sattributes:
    passthrough: false
    extract:
      metadata: [k8s.pod.name, k8s.namespace.name, k8s.node.name, k8s.deployment.name]
  resourcedetection:
    detectors: [env, system, eks]
  batch:
    send_batch_size: 8192
    timeout: 5s
exporters:
  otlp/gateway:
    endpoint: otel-gateway.observability:4317
    tls: { insecure: true }
    sending_queue: { enabled: true, queue_size: 5000 }
    retry_on_failure: { enabled: true, max_elapsed_time: 120s }
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, resourcedetection, batch]
      exporters: [otlp/gateway]
# gateway.yaml — Deployment, scaled by ingest volume. Big, stateful decisions.
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 6000             # tail sampling needs room
    spike_limit_mib: 1000
  tail_sampling:
    decision_wait: 30s
    num_traces: 100000
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 2 }
  transform/redact:
    error_mode: ignore
    trace_statements:
      - context: span
        statements:
          - replace_pattern(attributes["db.query.text"], "'[^']*'", "'?'")
              where attributes["db.query.text"] != nil
  batch: {}
exporters:
  otlp/tempo:
    endpoint: tempo-distributor:4317
    sending_queue: { enabled: true, queue_size: 20000 }
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, transform/redact, batch]
      exporters: [otlp/tempo]

Choosing a topology

Three topologies compared Three rows. Agent only has low cost, a single-node blast radius, no tail sampling and low complexity. Gateway only has medium cost, a fleet-wide blast radius, supports tail sampling and has medium complexity. Agent plus gateway has the highest cost, a single-node blast radius for ingest, supports tail sampling and has the highest complexity. What each topology costs and permits topology cost blast radius tail sampling complexity agent only low one node no low gateway only medium whole fleet yes medium agent + gateway highest one node yes highest Start with agents. Add a gateway when a requirement genuinely needs fleet-wide visibility — not before. The agent tier is what keeps a gateway restart from costing every service its in-flight spans.

The blast-radius column is the argument that usually settles it. With agents in front, a gateway rolling restart is absorbed by the agents’ retry queues and nothing is lost. Without them, every service’s exporter queue fills simultaneously and each one starts dropping spans independently.

Operating two tiers

Adding a gateway tier is not only an architecture decision; it is an operational commitment, and the day-two work is what determines whether it was worth it.

Scaling. The agent tier scales with your cluster automatically — it is a DaemonSet, one per node — while the gateway scales with ingest volume and must be sized deliberately. The signal to scale the gateway is sustained memory pressure or a rising receiver refusal rate, not CPU, because the gateway’s expensive processors are memory-bound. Tail sampling in particular has a memory requirement that grows with traffic and with decision wait, so a traffic increase can require a gateway scale-up long before anything looks CPU-constrained.

Upgrades. With agents in front, a gateway upgrade is a rolling restart absorbed by agent retry queues, and nothing is lost provided those queues are sized for the restart duration. Without agents, the same upgrade drops in-flight spans across every service simultaneously. That difference is the strongest practical argument for keeping the agent tier even when a gateway alone would technically suffice.

Configuration ownership. The two tiers change for different reasons and at different rates. Agent configuration changes when the platform changes — a new resource detector, a Kubernetes upgrade — and applies fleet-wide. Gateway configuration changes when policy changes: a new sampling rule, a new redaction pattern, a new tenant. Keeping them in separate files with separate review paths avoids the situation where a sampling tweak requires touching a DaemonSet that runs on every node in the cluster.

Failure isolation. Decide in advance what should happen when the gateway is unavailable. The default — agents queue, retry, and eventually drop — is usually right. The alternative, having agents fall back to exporting directly to the backend, sounds attractive and silently bypasses every gateway processor, which means redaction and sampling stop applying at exactly the moment nobody is watching. If you configure a fallback, make sure it is one you would be comfortable with running for a week.

Cost. Two tiers means paying for both. The agent tier is cheap per node and expensive in aggregate on a large cluster; the gateway tier is a handful of larger pods. A useful check is whether the gateway is doing something the agents genuinely cannot: if its configuration contains only batch and an exporter, it is a network hop with a monthly bill, and the agents can export directly.

Verification

# Agent tier: queue should stay near zero; a rising queue means the gateway is slow
otelcol_exporter_queue_size{job="otel-agent"}

# Gateway tier: refused spans indicate the memory limiter is shedding
rate(otelcol_receiver_refused_spans{job="otel-gateway"}[5m])

# Both tiers: every span accepted by an agent should reach the gateway
sum(rate(otelcol_receiver_accepted_spans{job="otel-agent"}[5m]))
  - sum(rate(otelcol_receiver_accepted_spans{job="otel-gateway"}[5m]))

A persistent gap in the last query means spans are being dropped between tiers — usually an undersized agent queue during gateway restarts.

Spans lost during a gateway rolling restart Spans lost during a gateway rolling restart. agents in front, queue sized 0 — absorbed; agents in front, small queue ≈40k spans; no agent tier ≈310k spans Spans lost during a gateway rolling restart agents in front, queue sized 0 — absorbed agents in front, small queue ≈40k spans no agent tier ≈310k spans The agent tier's queue is what turns a routine gateway upgrade into a non-event.

Common pitfalls

  • Running tail sampling on agents. Each agent sees a fraction of each trace, so decisions are made on incomplete data. It must run where whole traces arrive.
  • Sizing the agent like a gateway. An agent shares node resources with your workloads; a 4 GB memory limit on a DaemonSet is a capacity problem across the whole cluster.
  • Skipping the memory limiter on either tier. It is the only thing between a traffic spike and an OOM kill that takes the pipeline down.
  • Pointing SDKs at the gateway directly “for simplicity”. It works until the gateway restarts, at which point every service’s queue overflows at once.
  • Forgetting that k8sattributes needs the pod’s IP. Behind a gateway, the source IP is the agent, so pod enrichment must happen at the agent tier or it silently produces nothing.

Sidecar as a third option

Between a per-node agent and a central gateway sits the sidecar: one Collector container per application pod. It is worth knowing because it solves a specific problem well and is otherwise a poor default.

The problem it solves is isolation. A sidecar’s resource limits belong to the pod, so a service producing enormous span volume cannot starve its neighbours the way it can with a shared node-level agent. It also lets a team own its own Collector configuration without touching a cluster-wide DaemonSet, which matters in organisations where the platform team and the application teams move at different speeds.

The costs are real. One Collector per pod means hundreds of Collector processes, each with its own memory floor of roughly 50–100 MB, which on a large cluster is a substantial and largely idle allocation. Configuration drift becomes likely, because there is no single place to change a setting. And the sidecar still cannot do anything that needs fleet-wide visibility, so a gateway tier is usually needed anyway.

The pragmatic rule: use a node agent by default, a sidecar for the one or two services whose volume genuinely needs isolating, and a gateway when a requirement demands the whole picture. Mixing the three is normal, and the agent and sidecar can coexist as long as each service points at exactly one of them.

Related

↑ Back to OpenTelemetry Collector Pipeline Configuration