Measuring Baggage Header Overhead at the Edge

Log $http_baggage length at the ingress and compare it against the request payload: baggage under a few hundred bytes is negligible against a typical JSON body, but the same header is paid again on every internal hop, so a 600-byte baggage in a 12-hop request path costs 7 KB of internal traffic per request.

Context and when it matters

Baggage is convenient enough that it grows quietly. Someone adds a tenant ID, someone else a feature-flag cohort, a third team a locale and a currency, and each addition is individually trivial. Nobody measures the total, and the first sign of a problem is a 431 response or a truncated header at a proxy that nobody knew had a limit.

Measuring is cheap and worth doing before that point. The number also settles arguments: “baggage is expensive” and “baggage is free” are both wrong in most systems, and the specific figure for your request path decides which policy is right.

What the overhead actually is

Measuring it

At the ingress

# Log baggage and traceparent lengths alongside the request size.
log_format tracing '$remote_addr "$request" $status '
                   'req_bytes=$request_length resp_bytes=$bytes_sent '
                   'baggage_len=$http_baggage_length '
                   'hdr_bytes=$request_length';

# nginx has no direct length variable, so compute it in Lua (OpenResty)
# or approximate with the raw header in a debug-only log:
map $http_baggage $http_baggage_length {
    default 0;
    "~^(?<b>.*)$" $b;    # log the value; post-process for length
}

access_log /var/log/nginx/tracing.log tracing;

# Post-process: distribution of baggage sizes over an hour
awk -F'baggage_len=' '{print $2}' /var/log/nginx/tracing.log \
  | awk '{print length($1)}' | sort -n \
  | awk '{a[NR]=$1} END {
      printf "p50=%d p95=%d p99=%d max=%d n=%d\n",
        a[int(NR*0.5)], a[int(NR*0.95)], a[int(NR*0.99)], a[NR], NR}'
# p50=118 p95=604 p99=1180 max=3902 n=284119

That max is the number that matters most. A p50 of 118 bytes is irrelevant; a maximum of 3,902 bytes means some request is carrying nearly 4 KB of baggage, and it is one growth spurt away from a proxy limit.

At the mesh

# Envoy — emit header size in access logs so the internal hops are measurable too
access_log:
  - name: envoy.access_loggers.stdout
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
      log_format:
        json_format:
          path: "%REQ(:PATH)%"
          baggage_bytes: "%REQ(BAGGAGE)%"
          request_bytes: "%BYTES_RECEIVED%"
          upstream: "%UPSTREAM_CLUSTER%"

The limits you are heading towards

Each component in the path has its own header limit, and they are not the same. The first one you exceed is the one that fails, so the effective ceiling is the minimum across the whole path — usually a component nobody remembers is there.

The dangerous entries are the ones that truncate rather than reject: a silently shortened baggage header produces a malformed value downstream, and the failure appears as an application bug in a service that never touched baggage. Full treatment of the failure modes in avoiding 431 request header too large errors.

Turning the measurement into a control

# Middleware: record baggage size as a bounded bucket on every span.
BUCKETS = [(256, "lt_256"), (512, "lt_512"), (1024, "lt_1k"),
           (2048, "lt_2k"), (4096, "lt_4k")]

def baggage_bucket(raw: str | None) -> str:
    n = len(raw or "")
    for limit, label in BUCKETS:
        if n < limit:
            return label
    return "gte_4k"

span.set_attribute("baggage.size.bucket", baggage_bucket(request.headers.get("baggage")))
span.set_attribute("baggage.entry_count", (raw or "").count(",") + 1 if raw else 0)

Bucketing keeps the attribute bounded so it can be a metric dimension, which is what makes the trend chart above possible without an expensive trace query.

Deciding what the number means

A measurement is only useful next to a policy, and three thresholds cover most systems.

Under 512 bytes at p99. No action. The header is smaller than a typical HTTP request line plus cookies, and no mainstream proxy is anywhere near a limit. Keep measuring, do nothing else.

Between 512 bytes and 2 KB at p99. Worth a review, not an intervention. At this size the header is a meaningful fraction of small internal messages, and you are close enough to the lower limits — gRPC metadata, some servlet containers — that a single new entry could cross one. The useful action is to enumerate what is in there and confirm each entry is still used, because baggage entries outlive the features that added them more reliably than almost anything else in a codebase.

Above 2 KB at p99, or above 4 KB at max. Act. Either the entries are too many or the values are too large, and both have a straightforward fix: move anything that only the originating service needs onto a span attribute instead of into baggage, and replace long values with short identifiers that services can resolve locally. A tenant name of forty characters can almost always be a tenant ID of eight.

There is a fourth case worth naming because it is the one that surprises people: a p99 that is fine and a maximum that is enormous. That usually means one code path — an admin action, a bulk import, a specific integration — adds entries the rest of the traffic does not. It will not show up in an average and it will be the request that hits the limit first. Alert on the maximum, not the mean.

Finally, remember that the cost is not only bytes. Every service parses the baggage header on every request, and while that parse is fast, it is not free at high request rates with many entries. If you are measuring anyway, measure the parse: it is usually negligible, and knowing that it is negligible ends the argument.

Who owns the number

Baggage grows because it has no owner: every team can add an entry and none is responsible for the total. The measurement above fixes that only if someone looks at it, so the practical step is to attach the number to something that already has an owner — a platform dashboard, a monthly review, or an automated comment on any pull request that touches baggage configuration.

The lightest version that works is a check in the deployment pipeline: fail the build if the configured entry list exceeds an agreed count or the measured p99 exceeds the warning threshold. That converts an unbounded shared resource into one with a budget, and budget conversations are a great deal easier to have before the limit is reached than during the incident that follows it.

Common pitfalls

• Measuring only at the edge. The internal cost is the edge cost multiplied by the hop count, and internal payloads are much smaller, so the ratio is far worse inside.
• Assuming one limit applies everywhere. The effective ceiling is the minimum across the path, and it is usually a component nobody thought about.
• Ignoring gRPC. Metadata limits are lower than HTTP header limits and fail with a different error, which makes the connection harder to spot.
• Recording the raw baggage value as an attribute. It contains exactly the high-cardinality values baggage exists to carry; record the size, not the content.
• Treating truncation as a baggage problem. A truncated header breaks the next service to parse it, so the error surfaces somewhere unrelated.

Related

• Baggage size limits and header constraints — the limits and the policy for staying under them
• Avoiding 431 request header too large errors — what happens when the measurement is ignored
• Baggage vs span attributes: when to use what — deciding what deserves to be in the header at all

↑ Back to Baggage Size Limits and Header Constraints