Filtering and Transforming Spans in the Collector

The OpenTelemetry Collector’s filter processor decides which spans to drop and the transform processor rewrites the spans you keep — both driven by OTTL, where the single most important detail is knowing whether a statement runs in span context or resource context.

Context & When It Matters

Two problems show up in every trace backend once instrumentation matures. First, noise: liveness and readiness probes generate thousands of identical spans per hour that carry no diagnostic value but consume storage and clutter search results. Second, sensitive data: SDKs and framework instrumentation happily capture authorization headers, raw SQL, and query-string parameters that may contain personal data, all of which land in span attributes. The filter and transform processors solve both at the pipeline level, so you fix them once in the Collector rather than re-patching every service. Both are expressed in OTTL, the OpenTelemetry Transformation Language, a small statement language for reading and mutating telemetry as it flows through a processor.

Filter Versus Transform: Which Processor for Which Job

The two processors overlap enough to confuse, but the rule is simple: filter decides whether a span exists, transform decides what a span that exists looks like. filter is all-or-nothing at span granularity — a condition either drops the whole span or leaves it untouched. transform never drops a span; it only reads and writes fields on the spans that remain. Reaching for transform to “empty out” an unwanted span still leaves an empty span in the trace, which is worse than either keeping or dropping it cleanly.

Because filter shrinks the stream, ordering it before transform means the more expensive rewrite work only runs on spans you are actually keeping. Both belong between memory_limiter and batch so that dropped and reshaped spans are settled before any tail-based sampling decision or export.

Minimal Working Configuration

The configuration below drops the two most common probe routes and redacts one sensitive header. Read it first, then the explanation follows.

processors:
  filter/drop_health:
    error_mode: ignore
    traces:
      span:
        # Drop the span if EITHER condition is true (OR semantics)
        - 'attributes["http.route"] == "/healthz"'
        - 'attributes["http.route"] == "/readyz"'
  transform/redact:
    error_mode: ignore
    trace_statements:
      - context: span
        statements:
          - delete_key(attributes, "http.request.header.authorization")

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, filter/drop_health, transform/redact, batch]
      exporters: [otlp]

The filter processor uses drop semantics with OR logic: a span is removed if any listed condition evaluates true. Each condition is an OTTL boolean expression evaluated in span context, where attributes[...] addresses that span’s own attributes. Here any span whose http.route is a probe endpoint is discarded before it reaches storage.

The transform processor is a list of statement groups, each pinned to a context. This group runs in context: span, so its statements mutate individual spans; delete_key removes the captured authorization header outright. Note the ordering in the pipeline — filter runs before transform so rewrite work is never wasted on spans that are about to be dropped, and both sit between memory_limiter and batch in the mandatory pipeline order.

Implementation

The production configuration extends both processors and annotates the OTTL. filter is often best expressed with the drop_health pattern above, but complex noise suppression combines conditions; transform handles renaming, hashing, and conditional redaction that protects data crossing a security boundary.

processors:
  filter/noise:
    error_mode: ignore   # a malformed span skips the condition, not the pipeline
    traces:
      span:
        # Exact-match probe routes — cheap, runs on every span
        - 'attributes["http.route"] == "/healthz"'
        - 'attributes["http.route"] == "/readyz"'
        # Drop synthetic monitoring traffic by user-agent
        - 'attributes["user_agent.original"] == "kube-probe/1.29"'
        # Regex only where equality cannot express the rule
        - 'IsMatch(attributes["url.path"], "^/internal/metrics.*")'

  transform/reshape:
    error_mode: ignore
    trace_statements:
      # --- span context: reads/writes individual span fields ---
      - context: span
        statements:
          # Set a low-cardinality span name: "GET /orders/{id}"
          - set(name, Concat([attributes["http.method"], attributes["http.route"]], " ")) where attributes["http.route"] != nil
          # Hash PII so it stays correlatable but not readable
          - set(attributes["enduser.id"], SHA256(attributes["enduser.id"])) where attributes["enduser.id"] != nil
          # Strip raw query text; keep only the operation
          - delete_key(attributes, "db.statement") where attributes["db.system"] != nil
          # Redact a header only on spans that crossed the edge
          - delete_key(attributes, "http.request.header.cookie") where attributes["span.kind"] == "SERVER"
      # --- resource context: reads/writes resource-level attributes ---
      - context: resource
        statements:
          # resource.attributes here, NOT attributes[...]
          - set(attributes["deployment.environment"], "prod") where attributes["deployment.environment"] == nil

The controlling distinction is context. In a span statement, attributes[...] and bare fields like name refer to the span; the resource is reachable only as resource.attributes[...]. In a resource statement, attributes[...] refers to the resource’s own attributes and there is no span in scope at all. Choosing the wrong block is the error that produces “unable to resolve attribute” at startup. The where clause guards each statement so it only runs on spans that actually have the field — calling SHA256 on a missing attribute would otherwise error, and error_mode: ignore would silently skip it, leaving the value unredacted. Hashing rather than deleting keeps a value like enduser.id usable for correlation while removing its readable form, the right trade-off for data governed by the PII rules of the security-boundaries guide. Because these mutations happen before export, the backend never stores the original — and any tail-based sampling policy downstream sees the reshaped span.

An OTTL Function Reference for Spans

The statements above lean on a small set of OTTL functions that cover the majority of real filtering and reshaping work. Knowing which one to reach for keeps rules readable and cheap.

• set(target, value) — assigns a value to a field or attribute. Used for renaming (set(name, ...)), stamping defaults, and overwriting with a hashed value. Pair it with where so it fires only on relevant spans.
• delete_key(attributes, "key") — removes a single attribute. This is the primary redaction primitive for data you never want stored, such as an authorization header or a raw cookie.
• keep_keys(attributes, ["a", "b"]) — the inverse allowlist: drops every attribute except the named ones. Useful when instrumentation attaches dozens of low-value keys and you want to retain a known-good few.
• replace_pattern(target, regex, replacement) — rewrites part of a string in place, for example masking a numeric ID embedded in a URL path without deleting the whole attribute.
• SHA256(value) / Hash helpers — one-way transforms that keep a value correlatable across spans while removing its readable form. This is the workhorse for pseudonymizing user identifiers.
• IsMatch(target, regex) — a boolean used inside filter conditions or where clauses. Powerful but the most expensive option, so anchor the pattern and prefer exact equality where the rule allows it.
• Concat([a, b], sep) — joins strings, the standard way to build a low-cardinality span name from a method and a route.

Every one of these runs inside a context block, and the context determines what attributes[...] resolves to. A statement group is either context: span, context: spanevent, or context: resource; the group header, not the function, selects the scope. Keeping each mutation in the narrowest context that owns its field avoids the resolution errors that dominate OTTL debugging.

Decision Criteria

Confirm each filter and transform rule against this checklist before shipping it:

• Is every dropped span a leaf? Filter probe and synthetic spans, never a parent that other spans reference — orphaning children breaks the span tree.
• Does each OTTL statement use the right context block? span for span fields, resource for resource attributes. A mismatch fails at startup.
• Is every mutating statement guarded by a where clause? Prevents errors on spans missing the target field and makes intent explicit.
• Are exact matches used before regex? Reserve IsMatch for rules equality cannot express, and anchor patterns to limit backtracking.
• Did you verify with a debug exporter? Add debug with verbosity: detailed and confirm the post-transform span shows the hashed, renamed, redacted result.

kubectl logs -l app=otel-collector | grep -A2 "enduser.id"
# the printed value should be a 64-char hex hash, not the raw id

Common Pitfalls

• Over-filtering that breaks traces. A condition written to drop “internal” spans can accidentally match a parent span, orphaning its subtree. Test filter conditions against real traces and confirm no dropped span has children still in the pipeline.
• Span vs resource context confusion. Reading attributes["service.name"] in a span block returns nil because service.name is a resource attribute; it must be resource.attributes["service.name"]. The reverse mistake — expecting span fields in a resource block — fails outright. Pin each statement to the context that owns the field it touches.
• Expensive regex on high-cardinality attributes. IsMatch runs on every span against the full attribute value. A pattern applied to url.path (high cardinality) is far costlier than an exact comparison on http.route (low cardinality). Match cheaply first and let those drops shrink the volume the regex must scan.

FAQ

Why does my OTTL statement error with ‘unable to resolve attribute’ on resource fields?

You are reading a resource attribute from inside a span context. attributes[...] in a span statement addresses span attributes; resource attributes must be read as resource.attributes[...]. Mixing the two, or using the wrong context block, is the most common OTTL authoring error.

Will filtering a span break the rest of the trace?

It can. Dropping a leaf span like a health check is safe, but dropping a parent span orphans its children, which then reference a missing span ID and render as a broken tree in the backend. Filter only spans that are safe to remove wholesale, typically self-contained probe or synthetic-traffic spans.

Is regex in the filter processor expensive?

Relative to exact string comparison, yes. Every span is evaluated against every condition, so a regex like IsMatch on a high-cardinality attribute runs on the full span volume. Prefer exact equality on low-cardinality keys such as http.route, and reserve regex for cases plain comparison cannot express.

Related

• OpenTelemetry Collector Pipeline Configuration — where filter and transform sit in the full processor chain.
• Configuring the Batch and Memory Limiter Processors — the processors that bracket filter and transform.
• Security Boundaries in Distributed Tracing — the PII classification that drives redaction and hashing rules.

↑ Back to OpenTelemetry Collector Pipeline Configuration