Modern Python services rarely fail inside a single synchronous HTTP handler. Most production latency spikes happen between a FastAPI endpoint, a PostgreSQL query, and an asynchronous background worker.

In this guide, we wire OpenTelemetry traces, structured logs, and custom latency histograms together in Python so every request carries a single trace_id end-to-end.

1. Install the OpenTelemetry SDK and Instrumentation

pip install fastapi uvicorn opentelemetry-api opentelemetry-sdk \ opentelemetry-exporter-otlp opentelemetry-instrumentation-fastapi 

2. Configure TracerProvider, MeterProvider, and OTLP Export

Create telemetry.py to export spans and metrics over gRPC to any OTLP collector (such as SigNoz or an OpenTelemetry Collector):

import os from opentelemetry import trace, metrics from opentelemetry.sdk.resources import Resource from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.metrics import MeterProvider from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter def setup_telemetry(service_name: str = "billing-api"): endpoint = os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317") resource = Resource.create({"service.name": service_name, "deployment.environment": "prod"}) tracer_provider = TracerProvider(resource=resource) tracer_provider.add_span_processor( BatchSpanProcessor(OTLPSpanExporter(endpoint=endpoint, insecure=True)) ) trace.set_tracer_provider(tracer_provider) metric_reader = PeriodicExportingMetricReader( OTLPMetricExporter(endpoint=endpoint, insecure=True), export_interval_millis=10000, ) metrics.set_meter_provider(MeterProvider(resource=resource, metric_readers=[metric_reader])) 

3. Propagating Trace Context into Background Tasks

When a FastAPI route enqueues work onto a queue or thread pool, copy the W3C traceparent carrier so worker spans attach to the parent HTTP trace instead of starting disconnected root traces:

from fastapi import FastAPI, BackgroundTasks from opentelemetry import trace, metrics from opentelemetry.propagate import inject, extract from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor app = FastAPI() setup_telemetry() FastAPIInstrumentor.instrument_app(app) tracer = trace.get_tracer("billing.worker") meter = metrics.get_meter("billing.metrics") job_latency = meter.create_histogram("worker.job.duration_ms", unit="ms") def run_invoice_job(order_id: str, carrier: dict): ctx = extract(carrier) with tracer.start_as_current_span("worker.generate_invoice", context=ctx) as span: span.set_attribute("order.id", order_id) # Simulated work + metric recording job_latency.record(42.5, {"job.type": "invoice"}) @app.post("/orders/{order_id}/invoice") def enqueue_invoice(order_id: str, bg: BackgroundTasks): carrier = {} inject(carrier) bg.add_task(run_invoice_job, order_id, carrier) return {"queued": True, "order_id": order_id} 

4. Key Takeaways

  1. Always bind BatchSpanProcessor rather than SimpleSpanProcessor in production to avoid blocking the FastAPI event loop on collector I/O.
  2. Propagate traceparent across queues and background workers via inject() / extract().
  3. Pair custom histograms with trace exemplars so p99 alerts jump straight to the slow span.