What Is OpenTelemetry and How Distributed Tracing Works

The short answer

OpenTelemetry (OTel) is an open standard plus a set of SDKs and tools for collecting telemetry: traces, metrics and logs. It is vendor-neutral: you instrument your code once and send the data to any backend that understands the OTLP protocol.

Distributed tracing answers one question: what happened to a single request as it travelled through the frontend, API, queues, databases and other services. Each request gets a trace ID, every step records its own timing, and the backend assembles the steps into one timeline.

Traces and spans

• A trace is the full story of one request, from the first entry point to the last database call.
• A span is one operation inside that trace: an HTTP handler, a SQL query, a call to an external API. A span has a name, start and end time, status and attributes (for example http.route or db.system).
• Spans form a tree: the root span is the incoming request, and child spans are the work it triggered.

Looking at the tree, you immediately see which step took most of the time or where the error occurred, instead of guessing from logs of five different services.

How context propagation works

For spans from different services to end up in one trace, the trace ID must travel with the request. This is context propagation.

1. Service A starts a trace and creates a span.
2. When calling service B, the SDK adds a header with the trace ID and the current span ID. The default format is W3C Trace Context (traceparent header).
3. Service B reads the header and creates its spans as children of A's span.
4. The same works for message queues: the context is put into message headers.

If one service in the chain does not forward the headers, the trace breaks into pieces. This is the most common reason for "incomplete" traces.

The OpenTelemetry Collector

The Collector is a separate process that sits between your applications and the backend. Its pipeline has three parts:

• receivers accept data (OTLP, Jaeger, Prometheus and others);
• processors batch, filter, enrich data and apply sampling;
• exporters send data to one or several backends.

A minimal configuration looks like this:

receivers:
  otlp:
    protocols:
      grpc:
      http:
processors:
  batch:
exporters:
  otlp:
    endpoint: tempo:4317
    tls:
      insecure: true
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

Why bother with a Collector instead of exporting directly from the app? You can change the backend without redeploying services, keep secrets out of application code, and control sampling and data volume in one place.

Getting started in practice

• Start with automatic instrumentation: for many languages there are agents or libraries that create spans for HTTP servers, clients and database drivers without code changes.
• Set a clear service.name for every service, otherwise the traces will be hard to read.
• Add manual spans only around important business operations.
• Configure sampling: storing every trace of a high-traffic system is expensive. Head sampling keeps a fixed share of traces; tail sampling in the Collector can keep all traces with errors or high latency.
• Link logs and traces by writing the trace ID into every log line.

Backends compared

Because OTel is vendor-neutral, the choice is not final: you can start with Jaeger or Tempo and later switch the exporter in the Collector.

Common mistakes

• Headers are not forwarded through a proxy, gateway or queue, so traces break.
• Too many spans with no value, for example a span for every small function.
• Personal data in span attributes: tokens, phone numbers, request bodies.
• No sampling, which leads to unexpected storage or SaaS bills.