A customer emails support: "Our invoice shows more AI tokens than our own dashboard does." You open the logs. There are millions of lines from the Laravel API, the Node.js billing worker and the queue, none of them share an ID, and the one request that matters was probably dropped by the 1% head sampling you turned on last quarter to cut the tracing bill. Nothing is broken. You just cannot see anything.

I wrote this with help from AI tools and checked every command, config key and claim myself against the current OpenTelemetry docs. I run Loop Origin, where we handle server setup and maintenance for Laravel and Node.js SaaS platforms, and this is the observability setup I reach for when the budget is small and the incidents are not: OpenTelemetry everywhere, logs that carry trace IDs, and a collector that keeps the traces worth paying for.

📚 Series Roadmap — Production AI & SaaS Systems Engineering

How do you run OpenTelemetry in a Laravel and Node.js SaaS without a huge bill?

Direct answer: Instrument every service with the OpenTelemetry SDK and send 100% of spans to your own OpenTelemetry Collector. Generate request-rate, error and latency metrics there from all spans, then use tail sampling to keep every error, every slow trace and a small random baseline. Put trace IDs in JSON logs so logs and traces join.

Why head sampling fails the incidents you care about

Head sampling decides at the first span whether a trace is recorded, before anyone knows how the request will end. At 1%, you keep 1% of your errors and 1% of your slow checkouts. That is fine for a high-traffic homepage and useless for a billing worker that fails twice a day.

Tail sampling flips the order. The collector buffers all spans of a trace for a few seconds, waits for the trace to finish, then decides. The cost moves from your tracing vendor's storage bill to a little collector memory, which is usually the cheaper side of the trade.

The catch: every span of a trace has to reach the same collector instance, or each instance sees half a trace and makes half a decision. That shapes the whole architecture below.

A steel gold panning pan resting on wet river stones with gravel swirling around a few gold nuggets, representing tail sampling that keeps only the valuable traces, AI-generated
AI-generated image.

The architecture 

 Laravel API ─┐ ┌─► tier-2 collector A ─┐ Node worker ─┼─► tier-1 collector(s) ──┤ (spanmetrics + ├─► traces backend Queue jobs ─┘ loadbalancing by └─► tier-2 collector B ─┘ metrics backend traceID tail_sampling)

On a small platform, tier 1 can be a collector sidecar or a single gateway; tier 2 can start as one instance. Split them when one box can no longer hold the decision buffer.

Tier 1: route by trace ID 

# otel-tier1.yaml (opentelemetry-collector-contrib) receivers: otlp: protocols: grpc: { endpoint: 0.0.0.0:4317 } http: { endpoint: 0.0.0.0:4318 } processors: memory_limiter: check_interval: 1s limit_percentage: 80 spike_limit_percentage: 20 exporters: loadbalancing: routing_key: traceID protocol: otlp: tls: { insecure: true } # private network only; use TLS across hosts resolver: dns: hostname: otel-tier2.internal port: 4317 service: pipelines: traces: receivers: [otlp] processors: [memory_limiter] exporters: [loadbalancing]

Tier 2: metrics from everything, traces from the interesting part 

# otel-tier2.yaml receivers: otlp: protocols: grpc: { endpoint: 0.0.0.0:4317 } connectors: spanmetrics: dimensions: - name: http.route - name: tenant.plan # low-cardinality only; never tenant.id forward: {} processors: memory_limiter: check_interval: 1s limit_percentage: 80 spike_limit_percentage: 20 filter/health: error_mode: ignore traces: span: - 'attributes["url.path"] == "/up"' tail_sampling: decision_wait: 10s num_traces: 50000 expected_new_traces_per_sec: 200 policies: - name: keep-errors type: status_code status_code: { status_codes: [ERROR] } - name: keep-slow type: latency latency: { threshold_ms: 1500 } - name: keep-billing type: string_attribute string_attribute: key: app.flow values: [billing, metering] - name: baseline type: probabilistic probabilistic: { sampling_percentage: 5 } batch: {} exporters: otlphttp/traces: endpoint: https://traces.example.internal otlphttp/metrics: endpoint: https://metrics.example.internal service: pipelines: traces/in: receivers: [otlp] processors: [memory_limiter, filter/health] exporters: [spanmetrics, forward] traces/sampled: receivers: [forward] processors: [tail_sampling, batch] exporters: [otlphttp/traces] metrics: receivers: [spanmetrics] processors: [batch] exporters: [otlphttp/metrics]

Three decisions matter here:

  • spanmetrics runs before sampling. Your request-rate, error and duration metrics are computed from every span, so dashboards and alerts stay accurate even though you store a fraction of the traces. This is what makes aggressive sampling safe.
  • Policies are OR'ed. A trace is kept if any policy says so: every error, every request over 1.5 s, every billing flow, plus 5% of everything else for "what does normal look like?" comparisons.
  • decision_wait must exceed your longest normal trace. Spans that arrive after their trace's decision can be evaluated as a new trace unless the processor's decision cache is configured, and you end up with fragments. Size num_traces for roughly decision_wait × traces per second, with headroom, and watch collector memory.

The health-check filter is a cheap win. Load balancer probes hitting Laravel's /up route every few seconds produce a lot of perfectly boring spans. Check which attribute your instrumentation actually emits (url.path in current semantic conventions, older libraries may use http.target) before you trust the filter.

Instrumenting Laravel

The PHP SDK uses a C extension for automatic hooks plus Composer packages:

pecl install opentelemetry composer require open-telemetry/sdk open-telemetry/exporter-otlp \ open-telemetry/opentelemetry-auto-laravel php-http/guzzle7-adapter 
; php.ini / FPM pool env extension=opentelemetry.so 
# .env or FPM pool OTEL_PHP_AUTOLOAD_ENABLED=true OTEL_SERVICE_NAME=api OTEL_TRACES_EXPORTER=otlp OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-tier1.internal:4318 OTEL_TRACES_SAMPLER=parentbased_always_on OTEL_PROPAGATORS=tracecontext,baggage

parentbased_always_on is deliberate: the SDK sends everything, the collector decides. Then make every log line carry the trace:

// app/Logging/AddTraceContext.php, registered with 'tap' => [AddTraceContext::class] namespace App\Logging; use Illuminate\Log\Logger; use Monolog\Formatter\JsonFormatter; use Monolog\LogRecord; use OpenTelemetry\API\Trace\Span; class AddTraceContext { public function __invoke(Logger $logger): void { foreach ($logger->getHandlers() as $handler) { $handler->setFormatter(new JsonFormatter()); $handler->pushProcessor(function (LogRecord $record): LogRecord { $ctx = Span::getCurrent()->getContext(); if ($ctx->isValid()) { $record->extra['trace_id'] = $ctx->getTraceId(); $record->extra['span_id'] = $ctx->getSpanId(); } return $record; }); } } }

Queued jobs lose context unless you carry it. When the API dispatches work to the Node.js billing worker from Part 4, I inject the W3C traceparent into the payload:

use OpenTelemetry\API\Trace\Propagation\TraceContextPropagator; $carrier = []; TraceContextPropagator::getInstance()->inject($carrier); Redis::rpush('usage:report', json_encode(['tenant_id' => $tenantId, 'otel' => $carrier]));

Instrumenting Node.js 

// instrumentation.ts, loaded with: node --require ./dist/instrumentation.js dist/worker.js import { NodeSDK } from '@opentelemetry/sdk-node'; import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node'; import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'; new NodeSDK({ serviceName: 'billing-worker', traceExporter: new OTLPTraceExporter(), // reads OTEL_EXPORTER_OTLP_ENDPOINT instrumentations: [getNodeAutoInstrumentations({ '@opentelemetry/instrumentation-fs': { enabled: false }, // noisy, rarely useful })], }).start(); 
// worker.ts: continue the trace started in Laravel import { context, propagation, trace, SpanStatusCode } from '@opentelemetry/api'; const tracer = trace.getTracer('billing-worker'); export async function handle(job: { tenant_id: number; otel?: Record<string, string> }) { const parent = propagation.extract(context.active(), job.otel ?? {}); return context.with(parent, () => tracer.startActiveSpan('usage.report', async (span) => { span.setAttribute('app.flow', 'billing'); try { await reportUsage(job.tenant_id); } catch (e) { span.recordException(e as Error); span.setStatus({ code: SpanStatusCode.ERROR }); throw e; } finally { span.end(); } })); }

The auto-instrumentation bundle includes Pino and Winston integrations that add trace_id and span_id to log records, so the same join works on both sides. Setting app.flow=billing is what lets the keep-billing policy keep the whole trace, including the Laravel request that started it. That answers the support email: one trace ID, from the API call to the Stripe meter event.

Choosing a sampling strategy

Strategy Where it decides Keeps all errors? Metrics accuracy Cost driver When I use it
No sampling Nowhere Yes Exact Backend storage grows with traffic Low traffic, early stage
Head sampling (e.g. 10%) SDK, at trace start No, ~10% Skewed unless metrics come from elsewhere Lower storage, blind to rare failures Very high traffic, uniform requests
Tail sampling + spanmetrics Collector, after trace ends Yes Exact (computed pre-sampling) Collector RAM, small storage Most B2B SaaS
Logs only, no tracing n/a n/a Depends on log parsing Log ingestion volume Never again, for me

Budget levers beyond sampling

  • Drop debug logs in production at the source, not in the pipeline; you pay to ship what you later delete.
  • Keep metric dimensions low-cardinality. A tenant.id dimension on spanmetrics creates one series per tenant per route. Put tenant IDs on spans, not on metrics.
  • Set retention per signal. Sampled traces for days, aggregated metrics for months, raw logs as short as your compliance allows.
  • Watch the collector itself. It exposes its own telemetry; alert on dropped spans and memory-limiter refusals.

Diagnostic checklist

  1. Can you go from any error log line to its full trace by copying trace_id?
  2. Does a trace started in Laravel continue through the queue into the Node.js worker?
  3. Are all spans of a trace routed to the same tail-sampling collector?
  4. Is decision_wait longer than your slowest normal trace?
  5. Are RED metrics generated before sampling?
  6. Does every policy you need (errors, latency, key flows) exist, plus a small baseline?
  7. Are health checks and other high-volume, low-value spans filtered?
  8. Are metric dimensions free of IDs, emails and URLs with parameters?
  9. Is the SDK sampler parentbased_always_on so the collector sees everything?
  10. Are collector drop and memory metrics alerting someone?

FAQ

Do I need a vendor to start?

No. The collector is open source and exports OTLP, so you can start with self-hosted backends such as Grafana Tempo, Jaeger or an OTLP-capable vendor's free tier, and switch later without touching application code.

How much memory does tail sampling need?

It depends on trace size, traffic and decision_wait. Measure it: run with memory_limiter, load-test at your peak traces per second, and size num_traces from what you observe rather than guessing.

Does sending 100% of spans to the collector slow PHP-FPM down?

The SDK batches exports, but every span still costs a little CPU in the request. Disable instrumentations you do not read, and keep the collector on the same host or network to keep export latency low.

Will tail sampling break my trace-based alerts?

It can if alerts count stored traces. Alert on the spanmetrics output instead, which sees every span.

How do I trace an LLM call inside a request?

Wrap it in a span with attributes for model, token counts and latency. Avoid storing prompts or outputs on spans; they can contain customer data and they inflate storage.

Is OpenTelemetry logging stable enough to replace my log shipper?

It is supported in many SDKs and the collector, but maturity differs by language. My conservative path is JSON logs with trace IDs to stdout, shipped as before, and moving to OTLP logs per service once its SDK is stable.

Next in this series, Part 6 covers LLM gateway and FinOps architecture: semantic caching, model routing and per-tenant token quotas, with the spans from this post as its cost ledger.

Follow & Connect