A single Collector samples without trouble, but as soon as replicas are added, decisions become inconsistent. Trace ID load balancing fixes this: a first tier of Collectors spreads spans by traceID, so every span of a trace lands on the same instance of the second tier, the one running the tail_sampling processor.

Goal: build this two-tier architecture with the load_balancing exporter from the OpenTelemetry Collector contrib repository, check it works and know its limits (pods added or removed, data loss). The principle is backend-independent: it applies to Dynatrace SaaS and Dynatrace Managed alike, because everything happens in the Collector, before the OTLP export.

This article complements the OpenTelemetry tail sampling tutorial, which covers sampling policies. Here the focus is the routing that makes them reliable.

Why trace ID load balancing is required for tail sampling

According to the tail_sampling processor README, all spans of a trace must be received by the same Collector instance for a correct decision. The processor keeps spans in memory, grouped by trace_id, then evaluates its policies. If two replicas share the spans of one trace, each decides on a partial view: one keeps the failing part, the other drops the rest. The result is truncated traces without a root span, which on-call engineers cannot use.

A classic network load balancer (round robin on port 4317) does not help: it spreads connections or requests, not traces. Deterministic routing on the trace identifier is needed. The exporter README describes exactly this use case: consistent routing that enables stateful downstream processing, tail sampling included.

How the load_balancing exporter works

The exporter reads the routing key (routing_key) of each batch, computes which backend it goes to and creates one OTLP exporter per endpoint. A few facts from its README:

  • Possible routing_key values are service, traceID, metric, resource, streamID, attributes and randomness. For traces, traceID is the default; stating it explicitly still helps readability.
  • The source of backends is a resolver: static (fixed list), dns (a name resolving to all addresses, for example a Kubernetes headless service), k8s (a Kubernetes service) or aws_cloud_map.
  • The backend choice depends on the identifier alone, not on actual load. The README states a distribution close to even, with a standard deviation under 5% in the current configuration.
  • Declared stability: beta for traces and logs, alpha for metrics.
  • The exporter type was renamed load_balancing; the old name loadbalancing is still accepted as a deprecated alias.

Here is the README excerpt showing the configuration structure, with the static resolver and, commented out, the DNS equivalent for Kubernetes:

Excerpt of the load\_balancing exporter README: routing\_key, OTLP protocol, static resolver and a commented dns resolver example using a headless service

Step-by-step setup

Step 1: the first tier, routing by traceID 

# collector-routing.yaml receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 exporters: load_balancing: routing_key: traceID protocol: otlp: timeout: 1s tls: insecure: true resolver: dns: hostname: otel-sampler-headless.observability.svc.cluster.local port: 4317 service: pipelines: traces: receivers: [otlp] exporters: [load_balancing]

Critical lines: routing_key: traceID guarantees one backend per trace; the dns resolver points at a headless service so that each second-tier pod appears as a distinct address. The DNS resolver re-queries the name periodically (5 s by default according to the README). tls.insecure: true suits a test; in production, use TLS between the two tiers.

If you prefer the k8s resolver, the README states that the first-tier pod must run with a service account allowed to get, list and watch EndpointSlice objects in the target namespace; otherwise the resolver cache stays empty and the exporter logs an error.

Step 2: the second tier, which samples 

# collector-sampler.yaml receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 processors: memory_limiter: check_interval: 1s limit_percentage: 80 spike_limit_percentage: 20 tail_sampling: decision_wait: 10s num_traces: 100000 policies: - name: errors type: status_code status_code: status_codes: [ERROR] - name: slow type: latency latency: threshold_ms: 1000 - name: normal-sample type: probabilistic probabilistic: sampling_percentage: 5 batch: {} exporters: otlphttp/dynatrace: endpoint: "https://<environment-id>.live.dynatrace.com/api/v2/otlp" headers: Authorization: "Api-Token ${env:DT_API_TOKEN}" service: pipelines: traces: receivers: [otlp] processors: [memory_limiter, tail_sampling, batch] exporters: [otlphttp/dynatrace]

Deploy this tier as a StatefulSet or Deployment behind the headless service named in step 1, with at least two replicas. The Dynatrace token needs the openTelemetryTrace.ingest scope; on Managed, the endpoint takes the form https://<your-domain>/e/<environment-id>/api/v2/otlp. For deploying the Collector on Kubernetes, see the OpenTelemetry Collector on Kubernetes.

Why two tiers rather than two pipelines in the same instance? The processor README says it is technically possible, but separating the layers gives better failure isolation. It describes the same architecture, one layer with the load balancing exporter and another with the tail sampling processor:

Excerpt of the tail\_sampling processor README: the section on scaling with two layers of collectors, the load balancing exporter then the tail sampling processor

Step 3: tune resilience

By default, according to the README, the exporter does not try to re-route data to a healthy backend on delivery failure: queue, retry and timeout are disabled at the exporter level itself. Two levels of settings exist:

  • Under protocol.otlp (options 2): timeout, retry_on_failure and sending_queue for temporary problems with one backend, such as a brief network glitch. Enabled by default.
  • Directly in the load_balancing section (options 1): the same settings, useful in an elastic environment such as Kubernetes where the endpoint list changes on every rollout. Disabled by default.

The README also states that data loss remains possible if all targets stay unavailable once retries are exhausted. Size the queues accordingly and alert on export failures.

What happens when the number of pods changes

When the backend list changes, part of the routes are reassigned: roughly R/N, where R is the number of routes (here, trace IDs) and N the number of backends. The more backends, the smaller the disruption. For a trace in flight during a rollout, its spans may therefore end up on two instances and produce an incomplete trace. The README notes that the dns and k8s resolvers reflect changes with a delay, the k8s resolver updating faster than dns.

What this means for a platform team:

  • Avoid restarting the whole second tier at once: roll out gradually and watch the rate of traces without a root span.
  • If the backend list changes constantly and stability matters, the README suggests considering the groupbytrace processor upstream, which dispatches traces atomically to the exporter.
  • Set a decision_wait longer than your longest traces. Decisions already taken can be kept by the decision caches (decision_cache), disabled by default.

How to check that it works

  1. A local test: the exporter README offers a configuration where the load balancer and four backends run on the same machine, each backend exporting to the debug exporter. Send multi-span traces and check that each traceID appears in a single backend only.
  2. First-tier logs: an error such as couldn’t find the exporter for the endpoint points, according to the README, to a k8s resolver lacking RBAC permissions.
  3. Collector internal metrics: on the second tier, follow the tail sampling processor counters. Exact names vary by version; list them on the internal telemetry endpoint rather than copying them from a tutorial.

  4. Dynatrace SaaS: look for traces without a root span, a sign of scattered spans or too short a decision_wait:

    fetch spans, from: now() - 1h | summarize spans = count(), roots = countIf(isNull(span.parent_id)), by: {trace.id} | filter roots == 0 | limit 20

  5. Dynatrace Managed: in the distributed traces view, filter on failed requests and check that traces are not truncated (DQL queries do not apply to Managed).

To correlate logs and traces after sampling, see OpenTelemetry logs and the trace ID. Routing by traceID also works for logs: the README states that logs without a trace ID are distributed randomly.

Best practices

  • Separate the two tiers: the first is stateless, the second holds traces in memory.
  • Use a headless service and a dns or k8s resolver rather than a static list, which exposes you to loss if a backend goes down.
  • Size second-tier memory from throughput, decision_wait and num_traces, and put memory_limiter first.
  • Compute RED metrics before sampling, on the first tier; with traceID routing, every Collector sees all service and operation combinations, which can cause label collisions (the README recommends service routing for that case).
  • Treat a truncated trace as a configuration defect, never as one team’s fault.

Frequently asked questions

Is trace ID load balancing needed with a single Collector?

No. With a single instance receiving every span, the processor sees each trace in full. Routing becomes mandatory as soon as several replicas sample.

What is the difference between routing_key traceID and service?

With traceID, all spans of a trace go to the same backend, which tail sampling requires. With service, all spans of a service go to the same backend, which suits metrics computed per service.

Is an ordinary Kubernetes load balancer enough?

No: a classic Service spreads connections, with no link to the traceID. You need the load_balancing exporter and a headless service so that each pod is individually addressable.

Conclusion

Tail sampling is only reliable at scale if every trace arrives whole on a single instance. The load_balancing exporter with routing_key: traceID is the condition, at the price of an extra Collector tier and some care during rollouts. Next, see context propagation with Kafka, which prevents other broken traces upstream. References: the load_balancing exporter README and the tail_sampling processor README.

span.observer is an independent community blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of its owner.

Originally published on span.observer. French version: https://span.observer/fr/load-balancing-trace-id-tail-sampling/.