Recording Exceptions and Error Status on Spans

Set StatusCode.ERROR only when the operation the span represents actually failed, pair it with a bounded error.type for grouping, and record the exception event once — at the point the exception is handled, not at every frame it passes through.

Context and when it matters

Error handling in traces goes wrong in two opposite directions, and both make the data useless.

The first is over-reporting. Every layer catches, records the exception, re-raises, and the next layer does the same. A single failure produces seven exception events across five spans, the trace shows a wall of red, and the error rate computed from span status counts one request as five failures. The second is under-reporting: a handler catches an exception, returns a 500, and never touches the span, so the trace shows a perfectly healthy request that happened to take 40 ms and return nothing.

Getting this right matters because span status is not decoration — it is the numerator of every error-rate metric derived from traces, the filter behind “show me failing traces”, and the input to trace-based alerting. See trace-based alerting and SLO monitoring for what consumes it downstream.

The three signals, and what each is for

OpenTelemetry gives you three distinct mechanisms, and they answer different questions.

Span status, error.type, and exception events Three columns. Span status is unset, ok, or error, is used for error rate and red highlighting, and is bounded to three values. The error.type attribute is a short taxonomy value used for grouping and alerting, and should have tens of values. The exception event carries type, message, and stack trace for forensics, is rarely indexed, and can be large. Different questions, different mechanisms Span status "did this operation fail?" UNSET · OK · ERROR drives error rate colours the waterfall exactly 3 values error.type "what kind of failure?" timeout · 503 · deadlock groups failures safe as a dimension tens of values exception event "what exactly happened?" type · message · stack forensics only usually not indexed unbounded, expensive Set all three on a genuine failure. Set only the first two when you have a status code but no exception object.

The distinction that saves the most grief: error.type is a taxonomy, not a message. "connection reset by peer while reading from 10.2.4.19:5432" is a message. "connection_reset" is a taxonomy value. The first has one value per occurrence and cannot be grouped; the second has a handful of values and makes “which failure mode is dominant right now” a one-line query.

Implementation

Decide whose fault it is

The convention is that span status reflects the operation the span represents, judged from the perspective of the code that created it.

# A 404 is a correct answer to a request for something that does not exist.
# The server span is NOT an error; the operation succeeded.
if not order:
    span.set_attribute("http.response.status_code", 404)
    # status stays UNSET — this request did what it was supposed to do
    return JSONResponse({"error": "not found"}, status_code=404)

# A 500 means this service failed at its job.
except DatabaseUnavailable as exc:
    span.set_attribute("http.response.status_code", 500)
    span.set_attribute("error.type", "database_unavailable")   # taxonomy
    span.set_status(Status(StatusCode.ERROR, "orders database unreachable"))
    span.record_exception(exc)          # once, here, at the handling boundary
    return JSONResponse({"error": "internal"}, status_code=500)

There is a nuance on the client side. A CLIENT span that receives a 404 from a peer is an error for the client span if the client expected the resource to exist — the operation “fetch order 42” failed. The same 404 on the peer’s SERVER span is not. Two spans, same HTTP response, different statuses, and both are correct.

Record the exception once

The recording point is where the exception is handled, not where it is observed passing by. Recording at every frame produces a duplicate event on every span in the stack.

// Wrong: every layer records, so one failure becomes five exception events.
async function loadOrder(id) {
  const span = tracer.startSpan('loadOrder');
  try { return await repo.fetch(id); }
  catch (err) { span.recordException(err); throw err; }   // recorded here...
  finally { span.end(); }
}

// Right: intermediate layers mark status, the handler records the exception.
async function loadOrder(id) {
  const span = tracer.startSpan('loadOrder');
  try {
    return await repo.fetch(id);
  } catch (err) {
    // Status yes — this span did fail — but no duplicate exception event.
    span.setAttribute('error.type', err.constructor.name);
    span.setStatus({ code: SpanStatusCode.ERROR });
    throw err;
  } finally {
    span.end();
  }
}
// ...and exactly one recordException() in the outermost handler that swallows it.

Most SDKs make this easy by recording the exception automatically when an exception escapes a start_active_span block, which is precisely the “handled at the boundary” semantics. If you rely on that, do not also call record_exception by hand — that is the most common source of duplicates.

Keep stack traces proportionate

An exception event with a full stack trace is 2–8 KB. On a rare failure that is a bargain. On a failure occurring 500 times a second during an incident, it is 4 MB/s of export traffic added exactly when the pipeline is already under pressure — and the collector’s memory limiter will start shedding spans in response.

# Record the full stack only for unexpected failures; keep known,
# high-frequency failures cheap.
KNOWN_FAILURES = {"timeout", "rate_limited", "connection_reset"}

def record_failure(span, exc, kind: str):
    span.set_attribute("error.type", kind)
    span.set_status(Status(StatusCode.ERROR))
    if kind not in KNOWN_FAILURES:
        # escape_hatch: full detail for the things we do not understand yet
        span.record_exception(exc, escaped=True)
    else:
        # a known failure needs a count, not a novel
        span.add_event("failure", {"error.type": kind})

Errors that cross a service boundary

A failure rarely stays in one process, and the convention has a specific answer for what each side records. The service that failed records the exception and sets ERROR. The caller sees a 500 or a gRPC UNAVAILABLE, and its CLIENT span is also an error — but the caller has no access to the callee’s exception and should not invent one. It records error.type from what it can observe: the status code, the gRPC code, or timeout if nothing came back at all.

This produces a useful pattern in the waterfall: a chain of ERROR spans whose error.type values change at the boundary. connection_reset at the database, database_unavailable in the repository that caught it, 500 in the calling service’s client span, 500 at the gateway. Reading that chain from the bottom up tells you both the root cause and how each layer interpreted it, which is frequently where the real bug lives — a service that maps a transient timeout to a permanent failure will show timeout at one level and invalid_request at the next, and that mismatch is the finding.

Retries deserve explicit attention here because they are the most common source of misleading error rates. A resilient client that retries three times and succeeds has produced two failed CLIENT spans and one successful one; the user saw success. If your error-rate metric counts all spans rather than server spans, that request contributes two errors to a dashboard nobody can reconcile with the user experience. Filter error rates to span.kind = server, and record http.request.resend_count so the retry behaviour remains visible without polluting the headline number.

The same reasoning applies to circuit breakers and fallbacks. A call that failed but was served from a cache is an error on the client span and a success on the server span that returned the cached value — and the trace should show both, because “we were degraded but users did not notice” is a distinct operational state worth being able to query for.

Verification

Three checks confirm the semantics are right:

What correct error propagation looks like in a waterfall Four nested spans. The HTTP server span at the top is marked error with one exception event and a 500 status. The service span below it is marked error with no exception event. The repository span is marked error with no exception event. The database span at the bottom is marked error with error type connection reset. Only one exception event exists in the whole trace. Four failed spans · exactly one exception event POST /checkout · ERROR · 500 exception recorded here CheckoutService.place · ERROR status only OrderRepository.save · ERROR status only INSERT orders · ERROR error.type = connection_reset The deepest ERROR span with an error.type is the root cause; everything above it is propagation. 
# Error rate should count requests, not exception events
{ resource.service.name = "checkout-api" && span:kind = server && span:status = error }
  | by(span.error.type) | count()

# A healthy taxonomy has a handful of rows. Hundreds of rows means someone is
# putting a message into error.type.

If the error rate computed from server spans is several times the rate of failed requests measured at the load balancer, intermediate spans are being counted — filter to span:kind = server in the metric definition.

Decision criteria

  • Did the operation this span represents fail? Yes → ERROR. The caller’s expectations were not met by this span’s work.
  • Is the failure the client’s fault? A 4xx on a server span is normally UNSET. The exception is authentication and authorization failures if you treat them as security events worth surfacing.
  • Is this the layer that handles the exception? Only then call record_exception.
  • Is the value bounded? error.type must be. Derive it from the exception class or the status code, never from the message.
  • Will this failure be frequent? Then skip the stack trace and add a lightweight event instead.

As a matrix, for the cases that come up most:

Which signals to set, by outcome Five rows with three columns. A 404 on a server span sets none of the three. A 404 on a client span that expected the resource sets status and error type. An unhandled 500 sets all three. A timeout to a peer sets status and error type but no stack trace. A retried call that succeeded sets status and error type on the failed attempts only. Set only what the outcome justifies outcome status ERROR error.type exception 404 on the SERVER span — — — 404 on the CLIENT span that expected it yes yes — unhandled exception → 500 yes yes yes timeout calling a peer (frequent) yes yes event only retried, eventually succeeded failed tries only yes —

Common pitfalls

  • Using record_exception as a logger. It is not a log line; it is an expensive structured event. High-frequency calls to it are the fastest way to double your export volume during an incident.
  • Setting OK everywhere. OK is meant for explicit assertions of success and suppresses a backend’s ability to infer status. Leave successful spans UNSET unless you have a specific reason.
  • Putting the exception message in error.type. It looks helpful and destroys grouping — see controlling span attribute cardinality.
  • Marking retried operations as errors without saying so. A call that failed twice and succeeded on the third attempt should show two error spans and one success, with http.request.resend_count set — otherwise the error rate looks alarming for an outcome the user never noticed.
  • Recording exceptions on already-ended spans. The call silently does nothing; enrich before end().

Related

↑ Back to Span Attributes and Semantic Conventions