Instrumenting gRPC Services with OpenTelemetry

gRPC carries trace context in call metadata, not HTTP headers, so instrument both ends: attach the OpenTelemetry client instrumentation to inject traceparent into outgoing metadata, and the server instrumentation to extract it — in Python use GrpcInstrumentorClient/GrpcInstrumentorServer, in Go register otelgrpc.NewClientHandler/NewServerHandler as stats handlers.

Context and when it matters

gRPC multiplexes calls over HTTP/2 and passes per-call key/value pairs as metadata. That metadata is where the W3C TraceContext traceparent value must travel for a distributed trace to survive a hop between two services. Unlike a REST client, you cannot just set a header dictionary and forget it — the context propagation plumbing has to serialise the active context into the metadata map on the client and rebuild it on the server before the handler span opens.

gRPC also has four call shapes — unary, server-streaming, client-streaming, and bidirectional — and each affects how a span is scoped. A unary call maps cleanly to one client span and one server span. A streaming call keeps a single span open for the life of the stream. Getting that mapping right is why the choice between an interceptor and a stats handler matters, and why streaming RPCs are the classic place instrumentation goes wrong.

Where gRPC interceptors sit Where gRPC interceptors sit. stub call: client interceptor; metadata: context on the wire; server interceptor: extract; handler: business logic Where gRPC interceptors sit stub call client interceptor metadata context on the wire server interceptor extract handler business logic Interceptors are registered once per channel and per server — the handler never mentions tracing.

Minimal working code

Python — instrument both ends

The Python instrumentors patch the channel and the server so metadata carries context with no per-call code:

from opentelemetry import trace
from opentelemetry.propagate import set_global_textmap
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
from opentelemetry.instrumentation.grpc import (
    GrpcInstrumentorClient,
    GrpcInstrumentorServer,
)

# 1. Register a propagator so the SDK knows how to serialise context.
set_global_textmap(TraceContextTextMapPropagator())

# 2. Client side: injects traceparent into outgoing call metadata.
GrpcInstrumentorClient().instrument()

# 3. Server side: extracts traceparent from incoming metadata and
#    opens a SERVER span parented to the caller's span.
GrpcInstrumentorServer().instrument()

After this, a call from an instrumented client to an instrumented server shows one connected trace: the client CLIENT span, the traceparent value riding in metadata, and the server SERVER span pointing back to it as its parent.

Go — register stats handlers

In Go the current, correct approach is the stats handler, not the deprecated interceptors:

import (
    "google.golang.org/grpc"
    "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
)

// Server: the stats handler sees per-stream and per-message events,
// so streaming RPCs get correct span timing.
srv := grpc.NewServer(
    grpc.StatsHandler(otelgrpc.NewServerHandler()),
)

// Client: injects traceparent into the outgoing metadata on every call.
conn, err := grpc.NewClient(
    "inventory:50051",
    grpc.WithStatsHandler(otelgrpc.NewClientHandler()),
    grpc.WithTransportCredentials(insecure.NewCredentials()),
)

Verifying context flows through metadata

The proof that instrumentation works is a traceparent key appearing in the metadata the server receives. You can confirm this without a backend by reading metadata in a debug interceptor:

import grpc

class TraceparentProbe(grpc.ServerInterceptor):
    def intercept_service(self, continuation, handler_call_details):
        md = dict(handler_call_details.invocation_metadata)
        # A correctly instrumented client puts a W3C traceparent here.
        print("traceparent =", md.get("traceparent"))
        return continuation(handler_call_details)

A value shaped like 00-<32 hex trace-id>-<16 hex span-id>-01 confirms the client injected context and the server can extract it. An empty value means the propagator or client instrumentation is missing.

Implementation

The block below shows a hand-instrumented unary handler for the case where you need a child span around business logic inside an already-instrumented server. Each comment ties a line to a tracing concept.

from opentelemetry import trace, context
from opentelemetry.propagate import extract

tracer = trace.get_tracer("inventory-service", "2.1.0")

class InventoryServicer(inventory_pb2_grpc.InventoryServicer):

    def Reserve(self, request, grpc_context):
        # If you are NOT using GrpcInstrumentorServer, extract manually:
        incoming = dict(grpc_context.invocation_metadata())
        # extract() rebuilds the parent Context from the traceparent metadata.
        parent_ctx = extract(incoming)

        # start_as_current_span with the extracted parent links this
        # SERVER-side work to the caller's span across the process boundary.
        with tracer.start_as_current_span(
            "inventory.reserve", context=parent_ctx
        ) as span:
            span.set_attribute("sku", request.sku)
            span.set_attribute("rpc.system", "grpc")
            reserved = self._reserve(request.sku, request.qty)
            span.set_attribute("reserved", reserved)
            return inventory_pb2.ReserveReply(ok=reserved)

When the automatic GrpcInstrumentorServer is active you skip the manual extract() — the instrumentation has already opened the SERVER span and set the active context, so start_as_current_span("inventory.reserve") alone parents correctly.

RPC convention attributes and why each matters rpc.system groups by protocol *required rpc.service / rpc.method span identity required rpc.grpc.status_code error taxonomy *numeric, not the message server.address / port service map edges required on clients no url.* keys RPC has no URL common mistake RPC convention attributes and why each matters attribute purpose status rpc.system groups by protocol required rpc.service / rpc.method span identity required rpc.grpc.status_code error taxonomy numeric, not the message server.address / port service map edges required on clients no url.* keys RPC has no URL common mistake gRPC spans that borrow HTTP attribute names lose every RPC-specific backend view.

Decision criteria

gRPC status codes seen in one hour, by frequency gRPC status codes seen in one hour, by frequency. OK (0) 1.42M; DEADLINE_EXCEEDED (4) 3,100; UNAVAILABLE (14) 1,900; INVALID_ARGUMENT (3) 820 gRPC status codes seen in one hour, by frequency OK (0) 1.42M DEADLINE_EXCEEDED (4) 3,100 UNAVAILABLE (14) 1,900 INVALID_ARGUMENT (3) 820 Codes 4 and 14 are retry-and-capacity signals; code 3 is a client bug. Grouping by code separates them instantly.

Common pitfalls

  • Interceptor instead of stats handler for streaming. A unary interceptor fires once per call and cannot observe individual stream messages or the stream’s end, so a server-streaming RPC gets a span that closes at the wrong moment — or leaks open. Use the stats handler, which receives Begin/End and per-message events, so streaming spans have correct duration.
  • Metadata dropped alongside deadlines. gRPC deadlines and cancellations propagate through the same call context that carries metadata. If you rebuild a call with a fresh context.Context (Go) or a new metadata object to set a deadline, you can strip the injected traceparent unless you copy existing metadata forward. Merge, do not replace.
  • Manual and automatic instrumentation stacked. Registering both the automatic instrumentor and a hand-written tracing interceptor double-wraps each call, producing two SERVER spans. As with auto versus manual instrumentation generally, pick one boundary owner. gRPC over a service mesh adds a third potential span source, so agree on who owns the RPC span.

FAQ

Should I use a gRPC interceptor or a stats handler for tracing?

Prefer the stats handler where the SDK offers one. In Go, otelgrpc deprecated its interceptors in favour of NewClientHandler and NewServerHandler because the stats handler sees per-message and per-stream events that interceptors miss, producing correct spans for streaming RPCs.

Why is traceparent missing from my gRPC metadata?

The client-side instrumentation injects the traceparent key into the outgoing metadata map only when a global propagator is configured. If you never called set_global_textmap (Python) or otel.SetTextMapPropagator (Go), the SDK has no propagator to serialise context and the server extracts nothing.

How are spans created for a long-lived streaming RPC?

A single span covers the whole stream, opened when the RPC starts and closed when the stream terminates. Individual messages become span events rather than child spans, so a chatty stream does not explode into thousands of spans, but the parent span stays open for the stream’s full duration.

Streaming RPCs need a different span shape

Unary calls map cleanly onto one span. Streaming ones do not: a server-streaming RPC that pushes updates for an hour produces a span whose duration measures the subscription, not any unit of work, and a bidirectional stream has no natural single boundary at all.

The practical model is one span for the stream lifecycle carrying the connection-level attributes, plus a short span per message where the per-message work is significant. Where message volume makes that too expensive, record counts on the stream span instead — messages sent, messages received, and the final status — so the stream is at least summarisable even when individual messages are not traced.

↑ Back to Instrumenting Web Frameworks with OpenTelemetry