Observability for Async Work: Correlation IDs & Job Traces

A request returns 202, a job fails at 03:12, a webhook fires twice, and the only log line you have is processed invoice. Without a correlation ID that survives the hop from HTTP to queue to side effect, you are debugging with folklore. Async work is not "fire and forget". It is a distributed request that happens to pause in Redis, SQS or a database table.

I wrote this with help from AI tools and checked every Laravel, Node and OpenTelemetry example myself. I run Loop Origin, where we design and operate queue-heavy Laravel and Node systems for SaaS clients, and this is Part 5 of Reliable Queues & Async Work in SaaS. Part 1 covered idempotent jobs. Part 2 covered backpressure and tenant isolation. Part 3 covered dead-letter queues and safe replay. Part 4 covered delayed jobs and clock skew. Here we make the path from click to side effect visible.

📚 Series Roadmap — Reliable Queues & Async Work in SaaS

• Part 1: Idempotent Jobs & Exactly-Once Effects: Designing Safe Retries
• Part 2: Queue Backpressure, Fairness & Tenant Isolation in Multi-Tenant SaaS
• Part 3: Dead-Letter Queues, Poison Messages & Safe Replay Playbooks
• Part 4: Delayed Jobs, Schedulers & Clock Skew Gotchas
• Part 5 (This Post): Observability for Async Work: Correlation IDs & Job Traces
• Part 6: Choosing Redis, SQS or Database Queues for Laravel and Node SaaS

Building on: idempotency keys (Part 1), tenant-aware queues (Part 2), DLQ replay (Part 3), and due-at schedulers (Part 4).

How do you observe async jobs end-to-end?

Direct answer: Create a correlation ID (or W3C traceparent) on the inbound request, propagate it into every job payload and log context, start an OpenTelemetry consumer span when the worker runs, and emit structured fields that join request → job → side effect. Alert on age-in-queue and retry storms, not only on HTTP error rates.

Correlation ID vs trace ID vs job ID

You want all of them. Correlation IDs make grepping humane. Trace IDs make latency honest. Job IDs make replay safe. Idempotency keys make double-runs harmless.

Propagate through Laravel queues

// app/Http/Middleware/AssignCorrelationId.php public function handle($request, Closure $next) { $id = $request->headers->get('X-Correlation-Id') ?: (string) Str::uuid(); Context::add('correlation_id', $id); Log::withContext(['correlation_id' => $id]); $response = $next($request); return $response->header('X-Correlation-Id', $id); } 

// When dispatching — stash into the job + bus batch context ProcessInvoice::dispatch($invoice->id) ->withCorrelationId(Context::get('correlation_id')); // Job base trait public function middleware(): array { return [new RestoreCorrelationContext]; } public function handle(): void { Log::withContext([ 'correlation_id' => $this->correlationId, 'job_id' => $this->job?->getJobId(), 'invoice_id' => $this->invoiceId, ]); // ... work ... } 

Laravel 11's Context repository survives into queued jobs when you use the framework's context propagation; if you are on an older stack, serialize the correlation ID on the job explicitly. Either way, never rely on a static/global alone across worker processes.

Propagate through Node (BullMQ) with W3C traceparent

import { trace, context, propagation } from '@opentelemetry/api'; import { Queue, Worker } from 'bullmq'; const queue = new Queue('invoices'); export async function enqueueInvoice(invoiceId: string) { const carrier: Record<string, string> = {}; propagation.inject(context.active(), carrier); await queue.add('process', { invoiceId, correlationId: carrier['x-correlation-id'] ?? randomUUID(), traceparent: carrier['traceparent'], }); } const worker = new Worker('invoices', async (job) => { const parent = propagation.extract(context.active(), { traceparent: job.data.traceparent, }); const tracer = trace.getTracer('invoice-worker'); return context.with(parent, async () => { const span = tracer.startSpan('invoices.process'); try { span.setAttribute('invoice.id', job.data.invoiceId); span.setAttribute('correlation.id', job.data.correlationId); // side effects... } finally { span.end(); } }); }); 

The important part is propagation.inject at enqueue and extract at consume. Without that, every job starts an orphaned root span and your waterfall chart lies.

Structured logs that join the chain

Pick a flat JSON schema and refuse to deviate:

{ "ts": "2026-11-07T15:12:04.221Z", "level": "info", "msg": "invoice.webhook.sent", "correlation_id": "9f3c…", "trace_id": "4bf9…", "span_id": "a11c…", "job_id": "3129981", "tenant_id": "t_88", "invoice_id": "inv_1042", "attempt": 2, "queue": "invoices", "duration_ms": 184 } 

Now a support engineer can go from an HTTP access log to the webhook attempt without asking Slack. When we do Laravel development or server setup and maintenance work for clients, missing correlation fields are usually the first observability gap we close — before fancy dashboards.

Spans for the job lifecycle

Model four spans (or events) at minimum:

1. http.request — inbound API
2. queue.enqueue — attributes: queue name, delay, tenant
3. queue.consume — attributes: wait time, attempt, worker id
4. side_effect.* — outbound HTTP, email, ledger write

Wait time (consumed_at - enqueued_at) is the metric that catches the silent outages Part 2 and Part 4 warn about: a fair multi-tenant queue that is actually stuck, or a delayed job whose sweeper died.

# Example: p95 time-in-queue by queue name histogram_quantile(0.95, sum by (le, queue) (rate(queue_wait_seconds_bucket[5m]))) 

Dashboards and alerts that matter for async work

HTTP golden signals alone will stay green while invoices pile up. Instrument the broker path explicitly.

A before/after debugging story

Before: Support pastes an invoice ID. You grep six services for that ID, find three unrelated workers that happened to log it, and still cannot tell whether the webhook ran before or after the ledger write. Mean time to "we do not know" is about 40 minutes.

After: Support pastes the X-Correlation-Id from the API response (or from the customer-facing error page). One query across the log store returns, in order: http.request, queue.enqueue, queue.consume (attempt 2 after a timeout), side_effect.webhook.sent. The second attempt is visible as the same correlation ID with attempt: 2 and a new job_id — which is exactly what Part 1's idempotency key was protecting on the ledger side.

That is the bar. If a new hire cannot reconstruct that chain from docs and a single ID, instrumentation is incomplete.

Diagnostic checklist

1. Does every inbound request mint or accept a correlation ID and return it in the response?
2. Is that ID serialized onto every job payload (or restored via framework context)?
3. Do workers restore log context before the first log line of handle?
4. Are W3C traceparent (or equivalent) headers injected at enqueue and extracted at consume?
5. Does each job emit wait-time and attempt attributes on its consumer span?
6. Can you grep one correlation_id and see HTTP → job → webhook in order?
7. Do dashboards show oldest-message age, not only queue depth?
8. Are DLQ replays tagged with the original correlation ID plus a replay_of field?
9. Do scheduled/due-at jobs (Part 4) carry correlation or a deterministic idempotency key?
10. Have you tested killing a worker mid-job and confirming the retry keeps the same correlation ID?

FAQ

Is a UUID correlation ID enough without OpenTelemetry?

Yes for log joining and support. Add OTel when you need latency breakdowns across services. They complement; they do not replace each other.

Should the client send the correlation ID?

Accept an inbound X-Correlation-Id if present and valid; otherwise mint one. Never trust it as an auth token.

How do I handle fan-out (one request, many jobs)?

Keep one correlation ID for the user action; give each child job its own job ID and span with the same trace/correlation parent.

What about jobs scheduled days ahead?

Persist the correlation ID (or a business idempotency key) on the due-at row. Part 4's sweeper should copy it into the dispatched job unchanged.

Do I put PII in span attributes?

No. Prefer opaque IDs (invoice_id, tenant_id). Keep emails and names in redacted app logs if needed at all.

Where do I start if we have zero instrumentation today?

Correlation middleware + structured logs + oldest-message age alert. Spans can follow once grepping works.

Next in this series, Part 6 compares Redis, SQS and database queues for Laravel and Node SaaS: operational trade-offs, delay limits, fairness, and when a boring database table beats a clever broker.

Follow & Connect

• Website: Loop Origin
• LinkedIn: Loop Origin
• X: Muhammad Ali Mughal (@AliMughal887)
• GitHub: Loop Origin (@LoopOrigin)