Shadow Traffic and Dark Launch Routing with Baggage

Put a single shadow=true entry in baggage at the mirroring point, have every downstream service read it to suppress writes and tag its spans, and filter it out of every production dashboard — because the danger of shadow traffic is not that it fails but that it succeeds at something it should not have done.

Context and when it matters

Shadow traffic is the most honest test available: real requests, real data shapes, real concurrency, with no user affected by the result. It catches the class of problem that staging never does — a query that is fine against 10,000 rows and pathological against 40 million, a cache assumption that only breaks at production hit ratios, a serialization edge case that only real payloads contain.

It also has one catastrophic failure mode. A shadow request that charges a card, sends an email, or writes to the production database has caused a real incident from a test. Preventing that is not a routing problem but a context-propagation problem: every service in the path must know it is handling shadow traffic, including services three hops downstream that the mirroring proxy has never heard of. That is exactly what baggage is for.

The flag has to travel further than the mirror

The shadow flag must reach every downstream hop A gateway mirrors a request to both the production checkout service and a shadow checkout service. The shadow service calls the pricing and inventory services, which in turn call a database and a payment provider. The shadow entry in baggage reaches all of them, so writes are suppressed at every level rather than only at the mirrored service. baggage: shadow=true — carried to every hop, not just the first gateway mirrors 5% checkout v1 real · writes allowed checkout v2 shadow · writes off pricing reads shadow flag inventory reads shadow flag payment provider must not be called stock decrement must not happen A mirror that only marks the first hop leaves every service beyond it unable to tell shadow from real.

Implementation

Mark at the mirror

# Envoy — mirror 5% of traffic and mark the copy in baggage.
route:
  cluster: checkout-v1
  request_mirror_policies:
    - cluster: checkout-v2-shadow
      runtime_fraction:
        default_value: { numerator: 5, denominator: HUNDRED }
request_headers_to_add:
  - header:
      key: baggage
      value: "shadow=true,shadow.run=2026-08-05-a"
    append_action: APPEND_IF_EXISTS_OR_ADD

Appending rather than replacing matters: the request may already carry baggage entries that downstream services depend on, and overwriting them turns a shadow test into a behaviour change.

Suppress every side effect

from opentelemetry import baggage, trace

def is_shadow() -> bool:
    return baggage.get_baggage("shadow") == "true"

# One guard, applied at every boundary that leaves the process.
def guard_side_effect(name: str):
    if is_shadow():
        span = trace.get_current_span()
        span.add_event("side_effect.suppressed", {"side_effect.name": name})
        raise ShadowSuppressed(name)

async def charge_card(order):
    guard_side_effect("payment.charge")      # never reached on shadow traffic
    return await payments.charge(order)

async def decrement_stock(sku, qty):
    guard_side_effect("inventory.decrement")
    return await inventory.decrement(sku, qty)

Raising rather than silently returning is deliberate. A suppressed side effect changes the code path, and pretending it succeeded produces a shadow run whose results are not comparable to production. An explicit exception, caught and recorded at the top level, keeps the difference visible.

The alternative to guarding each call site — routing the shadow path to a separate database and a sandbox payment provider — is stronger where it is available, because it fails safe rather than relying on every developer remembering the guard. Use both if you can: separate infrastructure as the boundary, guards as the backstop.

Tag the spans

# Every span from a shadow request must be filterable out of production views.
if is_shadow():
    span.set_attribute("deployment.traffic_type", "shadow")
    span.set_attribute("shadow.run_id", baggage.get_baggage("shadow.run") or "unknown")
# Collector — route shadow traffic to its own tenant so it cannot contaminate
# production RED metrics or latency percentiles.
processors:
  routing:
    from_attribute: deployment.traffic_type
    default_exporters: [otlp/prod]
    table:
      - value: shadow
        exporters: [otlp/shadow]

This is the step teams skip, and the consequence is subtle: shadow traffic doubles the apparent request rate, adds a slower service’s latency into the aggregate, and makes an error-rate alert fire for failures no user experienced. Separating the streams keeps both datasets honest.

Compare the two paths

What a shadow run should report Four comparison rows over 41,000 mirrored requests. Response body matches in 99.2 percent of cases. The p95 latency is 210 milliseconds in production and 340 in shadow. The error rate is 0.1 percent in production and 0.4 percent in shadow. Suppressed side effects number 41,000, one per request, confirming the guard fired every time. Shadow run 2026-08-05-a · 41,000 mirrored requests measure production shadow verdict response body match — 99.2% investigate 0.8% p95 latency 210 ms 340 ms blocks release error rate 0.1% 0.4% investigate side effects suppressed n/a 41,000 guard held The last row is the safety check: one suppression per request means the flag reached every guarded call site.

A suppression count below the request count is the alarming result — it means some requests reached a guarded boundary without the flag, which is a propagation failure and a genuine risk. Alert on that ratio, not just on the comparison metrics.

Bounding the risk

Shadow traffic doubles load on every service in the path, so start at one percent rather than a round number that sounds harmless. Cap the run duration and make expiry automatic — a shadow run left on for a month becomes permanent infrastructure that nobody owns. And keep the flag out of anything a client can set: an externally supplied shadow=true would let a user request free checkouts, so strip and re-derive baggage at the ingress exactly as described in dynamic request routing with baggage.

Comparing responses without storing them

The most valuable output of a shadow run is a diff of what the two implementations returned, and the naive way to get it — log both responses and compare offline — creates a copy of production data in a place it does not belong.

The technique that avoids that is to compare in-process and record only the verdict. The mirroring layer, or a small comparison service, receives both responses, normalises them (sorting keys, dropping timestamps and generated identifiers), hashes each, and records whether the hashes matched plus a short classification of the difference when they did not. What lands in telemetry is response.match=false and response.diff_kind=field_missing, which is enough to investigate and contains no customer data.

Normalisation is where most of the effort goes, and it is worth being explicit about what is expected to differ. Timestamps, request IDs, and anything derived from wall-clock time or randomness will always differ and must be excluded, or the match rate is meaningless. Ordering differences in collections are usually acceptable and sometimes not — a sorted result set that becomes unsorted is a real regression, so decide deliberately rather than normalising it away.

The diff classification is what makes the result actionable. Grouping mismatches into a handful of kinds — a field missing, a value differing, a type changing, a collection of different length — turns “0.8 percent mismatch” into “0.8 percent mismatch, almost all of which are a field missing on one endpoint”, which is a bug report rather than a statistic.

One caution: a high match rate is weaker evidence than it appears. Shadow traffic exercises the paths production traffic happens to take, so a feature used by two percent of users contributes two percent of the comparison. If a specific behaviour matters, generate traffic that exercises it rather than assuming the mirror covered it.

Knowing when to stop

A shadow run needs an end condition agreed in advance, or it becomes permanent. Two conditions are usually enough: a volume target that gives statistical confidence — enough mirrored requests to have exercised the paths you care about — and a match rate with an agreed threshold. When both are met, the run ends and the mirror is removed.

The failure mode to avoid is the run that continues indefinitely because nobody wants to be the one to call it. Long-lived shadow traffic doubles cost, doubles load, and quietly becomes a dependency that the shadow service’s owners do not know they have. Put an expiry on the mirroring configuration itself, so continuing requires a deliberate act rather than inertia.

Load added to each downstream service by mirroring Load added to each downstream service by mirroring. mirror at 1% +1% everywhere; mirror at 5% +5% everywhere; mirror at 25% +25% everywhere; mirror at 50% +50% — capacity event Load added to each downstream service by mirroring mirror at 1% +1% everywhere mirror at 5% +5% everywhere mirror at 25% +25% everywhere mirror at 50% +50% — capacity event The load lands on every service in the path, including ones the shadow release does not touch.

Common pitfalls

  • Marking only the mirrored hop. Services further downstream cannot tell shadow from real and will happily perform writes.
  • Silently no-op’ing side effects. The shadow path then takes a different code path than production while appearing identical, and the comparison is worthless.
  • Letting shadow spans into production dashboards. Doubled request rate, inflated latency, spurious error alerts.
  • Trusting client-supplied baggage. A shadow flag settable from outside is an abuse vector, not a testing tool.
  • Forgetting the load. Mirroring at fifty percent adds fifty percent load to every service in the path, including the ones you were not testing.

Related

↑ Back to Dynamic Request Routing with Baggage