Instrumenting gRPC Services with OpenTelemetry

gRPC carries trace context in call metadata, not HTTP headers, so instrument both ends: attach the OpenTelemetry client instrumentation to inject traceparent into outgoing metadata, and the server instrumentation to extract it — in Python use GrpcInstrumentorClient/GrpcInstrumentorServer, in Go register otelgrpc.NewClientHandler/NewServerHandler as stats handlers.

Context and when it matters

gRPC multiplexes calls over HTTP/2 and passes per-call key/value pairs as metadata. That metadata is where the W3C TraceContext traceparent value must travel for a distributed trace to survive a hop between two services. Unlike a REST client, you cannot just set a header dictionary and forget it — the context propagation plumbing has to serialise the active context into the metadata map on the client and rebuild it on the server before the handler span opens.

gRPC also has four call shapes — unary, server-streaming, client-streaming, and bidirectional — and each affects how a span is scoped. A unary call maps cleanly to one client span and one server span. A streaming call keeps a single span open for the life of the stream. Getting that mapping right is why the choice between an interceptor and a stats handler matters, and why streaming RPCs are the classic place instrumentation goes wrong.

Minimal working code

Python — instrument both ends

The Python instrumentors patch the channel and the server so metadata carries context with no per-call code:

from opentelemetry import trace
from opentelemetry.propagate import set_global_textmap
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
from opentelemetry.instrumentation.grpc import (
    GrpcInstrumentorClient,
    GrpcInstrumentorServer,
)

# 1. Register a propagator so the SDK knows how to serialise context.
set_global_textmap(TraceContextTextMapPropagator())

# 2. Client side: injects traceparent into outgoing call metadata.
GrpcInstrumentorClient().instrument()

# 3. Server side: extracts traceparent from incoming metadata and
#    opens a SERVER span parented to the caller's span.
GrpcInstrumentorServer().instrument()

After this, a call from an instrumented client to an instrumented server shows one connected trace: the client CLIENT span, the traceparent value riding in metadata, and the server SERVER span pointing back to it as its parent.

Go — register stats handlers

In Go the current, correct approach is the stats handler, not the deprecated interceptors:

import (
    "google.golang.org/grpc"
    "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
)

// Server: the stats handler sees per-stream and per-message events,
// so streaming RPCs get correct span timing.
srv := grpc.NewServer(
    grpc.StatsHandler(otelgrpc.NewServerHandler()),
)

// Client: injects traceparent into the outgoing metadata on every call.
conn, err := grpc.NewClient(
    "inventory:50051",
    grpc.WithStatsHandler(otelgrpc.NewClientHandler()),
    grpc.WithTransportCredentials(insecure.NewCredentials()),
)

Verifying context flows through metadata

The proof that instrumentation works is a traceparent key appearing in the metadata the server receives. You can confirm this without a backend by reading metadata in a debug interceptor:

import grpc

class TraceparentProbe(grpc.ServerInterceptor):
    def intercept_service(self, continuation, handler_call_details):
        md = dict(handler_call_details.invocation_metadata)
        # A correctly instrumented client puts a W3C traceparent here.
        print("traceparent =", md.get("traceparent"))
        return continuation(handler_call_details)

A value shaped like 00-<32 hex trace-id>-<16 hex span-id>-01 confirms the client injected context and the server can extract it. An empty value means the propagator or client instrumentation is missing.

Implementation

The block below shows a hand-instrumented unary handler for the case where you need a child span around business logic inside an already-instrumented server. Each comment ties a line to a tracing concept.

from opentelemetry import trace, context
from opentelemetry.propagate import extract

tracer = trace.get_tracer("inventory-service", "2.1.0")

class InventoryServicer(inventory_pb2_grpc.InventoryServicer):

    def Reserve(self, request, grpc_context):
        # If you are NOT using GrpcInstrumentorServer, extract manually:
        incoming = dict(grpc_context.invocation_metadata())
        # extract() rebuilds the parent Context from the traceparent metadata.
        parent_ctx = extract(incoming)

        # start_as_current_span with the extracted parent links this
        # SERVER-side work to the caller's span across the process boundary.
        with tracer.start_as_current_span(
            "inventory.reserve", context=parent_ctx
        ) as span:
            span.set_attribute("sku", request.sku)
            span.set_attribute("rpc.system", "grpc")
            reserved = self._reserve(request.sku, request.qty)
            span.set_attribute("reserved", reserved)
            return inventory_pb2.ReserveReply(ok=reserved)

When the automatic GrpcInstrumentorServer is active you skip the manual extract() — the instrumentation has already opened the SERVER span and set the active context, so start_as_current_span("inventory.reserve") alone parents correctly.

Decision criteria

Common pitfalls

  • Interceptor instead of stats handler for streaming. A unary interceptor fires once per call and cannot observe individual stream messages or the stream’s end, so a server-streaming RPC gets a span that closes at the wrong moment — or leaks open. Use the stats handler, which receives Begin/End and per-message events, so streaming spans have correct duration.
  • Metadata dropped alongside deadlines. gRPC deadlines and cancellations propagate through the same call context that carries metadata. If you rebuild a call with a fresh context.Context (Go) or a new metadata object to set a deadline, you can strip the injected traceparent unless you copy existing metadata forward. Merge, do not replace.
  • Manual and automatic instrumentation stacked. Registering both the automatic instrumentor and a hand-written tracing interceptor double-wraps each call, producing two SERVER spans. As with auto versus manual instrumentation generally, pick one boundary owner. gRPC over a service mesh adds a third potential span source, so agree on who owns the RPC span.

FAQ

Should I use a gRPC interceptor or a stats handler for tracing?

Prefer the stats handler where the SDK offers one. In Go, otelgrpc deprecated its interceptors in favour of NewClientHandler and NewServerHandler because the stats handler sees per-message and per-stream events that interceptors miss, producing correct spans for streaming RPCs.

Why is traceparent missing from my gRPC metadata?

The client-side instrumentation injects the traceparent key into the outgoing metadata map only when a global propagator is configured. If you never called set_global_textmap (Python) or otel.SetTextMapPropagator (Go), the SDK has no propagator to serialise context and the server extracts nothing.

How are spans created for a long-lived streaming RPC?

A single span covers the whole stream, opened when the RPC starts and closed when the stream terminates. Individual messages become span events rather than child spans, so a chatty stream does not explode into thousands of spans, but the parent span stays open for the stream’s full duration.

↑ Back to Instrumenting Web Frameworks with OpenTelemetry