OpenTelemetry Instrumentation for Distributed Microservices Tracing

Beyond Basic Request Tracing

OpenTelemetry has become the standard for distributed tracing in microservice architectures, but most teams stop at basic request tracing — a span per HTTP call, maybe some database queries. That's like having security cameras that only record the lobby. You can see traffic entering and leaving, but you have no idea what's happening on each floor.

Effective instrumentation tells you why a request was slow, not just that it was. The difference is manual span creation at meaningful boundaries, propagated context that carries business metadata, and sampling strategies that capture the interesting traces without drowning in volume.

SDK Setup and Auto-Instrumentation

Start with auto-instrumentation. It handles HTTP clients, database drivers, gRPC, and message queues without code changes. Then layer manual instrumentation on top for business-specific spans.

# Python service setup
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor

resource = Resource.create({
    "service.name": "order-service",
    "service.version": "2.4.1",
    "deployment.environment": "production",
    "service.namespace": "commerce",
})

provider = TracerProvider(resource=resource)
processor = BatchSpanProcessor(
    OTLPSpanExporter(endpoint="otel-collector:4317"),
    max_queue_size=2048,
    max_export_batch_size=512,
    schedule_delay_millis=5000,
)
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

# Auto-instrument everything
FlaskInstrumentor().instrument()
SQLAlchemyInstrumentor().instrument()
RequestsInstrumentor().instrument()

The resource attributes are important. service.name is how your traces appear in the tracing backend. service.version lets you correlate performance changes with deployments. deployment.environment prevents you from accidentally mixing production and staging traces in the same view. I've seen teams skip these attributes and then struggle to filter traces when they need them during an incident.

Manual Instrumentation at Business Boundaries

Auto-instrumentation tells you that a database query took 200ms. Manual instrumentation tells you that the inventory check for order #12345 took 200ms because it queried 47 SKUs across 3 warehouses. The second version is dramatically more useful during troubleshooting.

tracer = trace.get_tracer("order-service.inventory")

def check_inventory(order):
    with tracer.start_as_current_span("inventory.check") as span:
        span.set_attribute("order.id", order.id)
        span.set_attribute("order.sku_count", len(order.items))

        results = {}
        for warehouse in get_active_warehouses():
            with tracer.start_as_current_span("inventory.warehouse_query") as wh_span:
                wh_span.set_attribute("warehouse.id", warehouse.id)
                wh_span.set_attribute("warehouse.region", warehouse.region)

                stock = warehouse.check_stock(order.items)
                wh_span.set_attribute("warehouse.items_available",
                    sum(1 for s in stock.values() if s > 0))

                results[warehouse.id] = stock

        span.set_attribute("inventory.warehouses_checked", len(results))
        span.set_attribute("inventory.fully_available",
            all(any(wh[sku] > 0 for wh in results.values())
                for sku in order.items))
        return results

The span attributes carry the business context that makes traces actionable. When someone reports that "inventory checks are slow for large orders," you can filter traces by order.sku_count > 20 and immediately see the latency distribution for large orders versus small ones. Without those attributes, you're guessing.

Context Propagation Across Service Boundaries

Tracing only works if context propagates from service to service. OpenTelemetry uses the W3C Trace Context standard by default — a traceparent header on HTTP requests and equivalent metadata for gRPC and messaging systems.

The auto-instrumentation libraries handle propagation for synchronous HTTP and gRPC calls automatically. Message queues require explicit handling because the "call" and the "response" happen at different times, often in different processes.

# Producer: inject context into message headers
from opentelemetry.propagate import inject

def publish_order_event(order):
    with tracer.start_as_current_span("order.publish") as span:
        span.set_attribute("order.id", order.id)

        headers = {}
        inject(headers)  # Injects traceparent into headers dict

        kafka_producer.send("orders",
            value=order.to_json(),
            headers=[(k, v.encode()) for k, v in headers.items()])

# Consumer: extract context from message headers
from opentelemetry.propagate import extract
from opentelemetry.context import attach, detach

def process_order_message(message):
    headers = {k: v.decode() for k, v in message.headers}
    ctx = extract(headers)
    token = attach(ctx)

    try:
        with tracer.start_as_current_span("order.process",
                context=ctx) as span:
            span.set_attribute("order.id", message.value["id"])
            # Process the order...
    finally:
        detach(token)

Without the extract/attach dance on the consumer side, the consumer creates a new trace instead of continuing the producer's trace. You end up with two disconnected traces — one for "order submitted" and one for "order processed" — with no way to correlate them. I've debugged this exact issue at least three times. It's subtle because the consumer still generates traces; they're just orphaned.

Sampling Strategies That Actually Work

At any meaningful request volume, tracing 100% of requests generates more data than you can store or afford. Sampling reduces volume while retaining enough traces for analysis and debugging.

Head-based sampling (decide at the start of a trace whether to record it) is simple but blind. You might sample out the one slow request per thousand that you actually need to investigate. Tail-based sampling (decide after the trace is complete) sees the full picture but requires buffering complete traces before the decision, which adds memory overhead and complexity.

Our collector configuration uses a probabilistic head sampler at 10% for normal traffic, with always-on sampling for errors and high-latency requests:

# otel-collector config
processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors-always
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow-always
        type: latency
        latency:
          threshold_ms: 2000
      - name: probabilistic-sample
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

This captures all errors, all slow requests, and a 10% sample of normal requests. The 10% sample gives us enough data for aggregate analysis (average latency, throughput trends, dependency maps). The always-on error and latency sampling ensures we never miss the traces that matter most during incident investigation.

Collector Architecture for Production

Don't send traces directly from your application to your tracing backend (Jaeger, Tempo, Datadog, etc.). Put an OpenTelemetry Collector between them. The Collector handles batching, retry, sampling, and routing — work you don't want your application doing on the hot path of request handling.

We run the Collector as a DaemonSet in Kubernetes (one per node) with a gateway Collector for cross-cluster aggregation. The DaemonSet Collector receives traces from pods on the same node over localhost (minimal network overhead), applies initial processing, and forwards to the gateway. The gateway applies tail sampling and exports to the storage backend.

This two-tier architecture keeps application-side latency minimal (traces go to localhost) while centralizing the expensive sampling and export logic in the gateway. The DaemonSet Collectors use about 256MB of memory each. The gateway needs more — typically 2-4GB depending on trace volume and tail sampling buffer duration.

One operational tip: set memory limits on the Collector containers and configure the memory limiter processor. Without it, a sudden traffic spike generates so many spans that the Collector OOM-kills itself, and you lose traces exactly when you need them most. The memory limiter drops spans when memory usage exceeds 80% of the limit — you lose some data, but the Collector stays alive.

Metrics and Logs in the OTel Pipeline

OpenTelemetry isn't just tracing. The same SDK and Collector handle metrics and logs, allowing you to correlate across all three signal types. A trace shows you that a specific request was slow. A metric shows you that the p99 latency increased at 2:30 PM. A log entry shows you the specific error message from the database driver. Linking them together is where observability becomes genuinely useful instead of just expensive data collection.

Exemplars are the bridge between metrics and traces. When you record a histogram observation (like request latency), you can attach a trace ID as an exemplar. Your metrics backend then lets you click through from a latency spike on the dashboard directly to a representative trace:

from opentelemetry import metrics

meter = metrics.get_meter("order-service")
latency_histogram = meter.create_histogram(
    "http.server.request.duration",
    unit="ms",
    description="Server request duration"
)

def handle_request(request):
    start = time.time()
    try:
        response = process(request)
        return response
    finally:
        duration_ms = (time.time() - start) * 1000
        latency_histogram.record(
            duration_ms,
            attributes={
                "http.method": request.method,
                "http.route": request.path,
                "http.status_code": response.status_code,
            }
        )

The attributes on the histogram observation become metric labels. This lets you filter the latency dashboard by route, method, or status code without creating separate histograms for each combination.

Log Correlation with Trace Context

Structured logging with trace context is the simplest form of correlation. When a log entry includes the trace ID and span ID, you can search logs for a specific trace and see exactly what happened during that request:

import logging
from opentelemetry import trace

class TraceContextFilter(logging.Filter):
    def filter(self, record):
        span = trace.get_current_span()
        ctx = span.get_span_context()
        record.trace_id = format(ctx.trace_id, '032x') if ctx.trace_id else ''
        record.span_id = format(ctx.span_id, '016x') if ctx.span_id else ''
        return True

logger = logging.getLogger(__name__)
logger.addFilter(TraceContextFilter())

handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter(
    '%(asctime)s %(levelname)s [trace=%(trace_id)s span=%(span_id)s] %(message)s'
))
logger.addHandler(handler)

Now every log line from within a traced request carries the trace ID. In Grafana, you can click from a Tempo trace to Loki logs filtered by that trace ID. In Datadog, the APM trace view shows correlated logs inline. The implementation cost is minimal — a logging filter and a format change — but the debugging value during incidents is substantial.

One operational note: log volume often exceeds trace volume by 10x or more. If your application generates chatty debug logs within traced spans, correlating them all can overwhelm your log storage. We filter log-to-trace correlation to WARNING level and above. Debug and info logs still get the trace ID in the log line for manual searching, but the tracing backend only links warning and error logs to their traces automatically.