Canary Routing with OpenTelemetry Baggage

Set a canary=true entry in OpenTelemetry baggage at ingress, derive an X-Canary header from it in middleware, and match that header in an Istio VirtualService — the flag then rides the baggage through every hop so the entire downstream call chain stays pinned to the canary deployment.

Context & When It Matters

A canary deployment is only trustworthy if a request that starts on the canary stays on the canary through every service it touches. If the checkout front end lands on v2-canary but the payment service it calls lands on v1-stable, you are testing a Frankenstein path that exists in neither the old nor the new release, and your canary metrics describe a system you will never ship. The goal of canary routing with baggage is cohort stickiness across the whole distributed trace: one routing decision at the edge, honoured everywhere downstream. This matters most for stateful or multi-hop flows — checkout, onboarding, anything where a mid-chain switch between builds corrupts the test.

Minimal Working Setup

Two pieces make this work: an ingress middleware that sets the flag, and a VirtualService that matches the derived header. Here is the smallest version of both.

# ingress: set canary baggage for a sticky cohort (FastAPI/Starlette)
from opentelemetry import baggage, context
import hashlib

CANARY_PERCENT = 10

async def canary_ingress(request, call_next):
    user_id = request.state.auth_claims["sub"]
    bucket = int(hashlib.sha256(user_id.encode()).hexdigest(), 16) % 100
    flag = "true" if bucket < CANARY_PERCENT else "false"

    token = context.attach(baggage.set_baggage("canary", flag))
    try:
        return await call_next(request)
    finally:
        context.detach(token)
# istio: route the canary cohort by the derived header
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: checkout-canary
spec:
  hosts:
    - checkout
  http:
    - match:
        - headers:
            x-canary:
              exact: "true"
      route:
        - destination: { host: checkout, subset: v2-canary }
    - route:   # unconditional fallback — everything else stays stable
        - destination: { host: checkout, subset: v1-stable }

The middleware writes canary=true into baggage only for the hashed cohort. A separate derivation step (shown next) turns that baggage entry into the X-Canary header the VirtualService matches. The final route has no match block, so any request without the derived header lands on v1-stable. That is the fallback the entire pattern depends on.

Implementation

The derivation step is what bridges baggage to the mesh. It reads the canary entry and emits the header only when the flag is genuinely true, so a false cohort produces no header and cleanly falls through to stable.

# derive_canary_header.py — baggage entry -> matchable header
from opentelemetry import baggage

def apply_canary_header(outbound_headers: dict, ctx=None) -> dict:
    # Read the canary flag that ingress set into baggage.
    flag = baggage.get_baggage("canary", context=ctx)

    # Emit X-Canary ONLY when true. A missing header is what routes
    # non-canary traffic to the stable fallback route in Istio.
    if flag == "true":
        outbound_headers["x-canary"] = "true"

    # NOTE: baggage itself is untouched — it keeps flowing to the next
    # hop so the canary flag survives the entire call chain, not just
    # this one outbound request.
    return outbound_headers

Bias sampling so canary traces are always kept. Without this, a low base sampling rate means most canary requests never produce a trace, and the regression you deployed the canary to catch stays invisible.

# canary_sampler.py — always record canary-flagged traces
from opentelemetry.sdk.trace.sampling import Sampler, SamplingResult, Decision
from opentelemetry import baggage

class CanaryBiasedSampler(Sampler):
    def __init__(self, base_ratio: float = 0.01):
        self._threshold = int(base_ratio * 0xFF)

    def should_sample(self, parent_context, trace_id, name, kind, attributes, links):
        # Canary requests are always recorded so regressions are visible.
        if baggage.get_baggage("canary", context=parent_context) == "true":
            return SamplingResult(Decision.RECORD_AND_SAMPLE)
        # Everyone else follows the base head-based rate.
        decision = Decision.RECORD_AND_SAMPLE if (trace_id & 0xFF) < self._threshold else Decision.DROP
        return SamplingResult(decision)

    def get_description(self):
        return "CanaryBiasedSampler"

This sampler leans on head-based sampling for the base cohort but overrides it to guarantee canary capture. The X-Canary derivation and VirtualService matching are the same mechanism described in the parent guide, dynamic request routing with baggage — this page narrows it to the canary case.

Verification Checklist

Work through these to confirm the cohort is genuinely pinned, not just intended:

  • Ingress sets the flag. Log the computed canary value at ingress and confirm the fraction of true matches your configured percentage across a sample of requests.
  • The header derives. Inspect an outbound request from the front end and confirm X-Canary: true is present for a canary user and absent for a stable user.
  • The mesh matched it. Grep the Envoy sidecar access log for x-canary and confirm the upstream_cluster resolved to the v2-canary subset.
  • The whole chain stayed on canary. Query Jaeger or Tempo for a canary-flagged trace and confirm every span reports the canary build version — no mid-chain switch to stable.
  • Stickiness holds. Replay several requests for the same user ID and confirm they all land the same way.

Common Pitfalls

  • Canary flag lost at an async hop. Baggage does not cross a Kafka topic or a gRPC stream by itself. A canary request that publishes to a queue and is consumed by a worker that never sees the flag will process on stable. Serialise baggage into message headers explicitly at the boundary — see propagating baggage across Kafka and gRPC — or accept that canary coverage ends at the queue.
  • No fallback route. If the VirtualService has only the x-canary match block and no unconditional final route, every non-canary request finds no matching route and Envoy returns 404 NR. The stable route must always exist as the last entry.
  • Sampling not biased to keep canary traces. A base rate of 1% applied uniformly means roughly one in a hundred canary requests produces a trace. The CanaryBiasedSampler above fixes this; skipping it leaves canary regressions statistically invisible.

FAQ

Why does a canary request drift back to stable partway through the call chain?

The canary baggage entry was lost at a hop. Most often a proxy stripped the baggage header, an async boundary dropped context, or a service failed to register the baggage propagator. Once canary is gone from baggage, the derived X-Canary header stops being emitted and the mesh routes the rest of the chain to stable.

Should canary assignment be random per request or sticky per user?

Sticky per user in almost all cases. Hash a stable identifier such as the user or account ID and compare it to the canary percentage. That keeps a given user consistently on canary or stable across requests, which makes their experience coherent and their traces comparable. Purely random per-request assignment fragments sessions across both builds.

How do I keep canary traces from being sampled away?

Read the canary baggage entry in a custom sampler and always record when it is true, regardless of the base sampling rate. Otherwise a 1% head-based rate means most canary requests produce no trace, and a rare canary regression may never appear in a sampled trace where you can diagnose it.

↑ Back to Dynamic Request Routing with Baggage