Distributed tracing

What a trace contains

A trace represents one unit of work, such as an HTTP request or an agent task. It is made of spans. Each span describes an operation, including its start and end time, name, status, attributes, and relationships to other spans. A database query, queue publish, model call, and downstream HTTP request may each appear as a span.

When spans are connected correctly, an investigator can see where time accumulated and where an error first appeared. For example, an API's slow response may be caused by a database call that waits behind a lock, or by a model request that retries after a provider timeout.

Propagate context consistently

Distributed tracing depends on carrying trace context across process and service boundaries. Instrumentation must preserve the parent and child relationship when work moves through an HTTP request, message queue, or asynchronous job. Missing propagation creates disconnected traces and makes a multi-service failure look like unrelated events.

Use stable names and carefully selected attributes. Avoid putting secrets or unbounded user content into telemetry. Sampling can reduce storage cost, but teams should understand what evidence is omitted and retain enough data to investigate important requests.

Traces support questions

Tracing is most valuable when it answers a concrete question: which dependency added latency, where did this request fail, or which model route produced the result? Pair traces with logs, metrics, deployment history, and outcome labels. That combination helps teams move from a symptom to a testable explanation without treating one timeline as the whole story.