Short answer: capture an API error as a small, typed incident record, then attach the experiment key, tenant cohort, request identifier, and trace context before you send it anywhere. Resolve it by comparing the same record across cohorts and by proving recovery with a later successful signal. A stack trace alone cannot reconstruct a marketplace rollout.

Here is the decision rule I use for a marketplace experiment: choose the event model that can answer “what did this tenant see, under which variant, and what happened next?” without joining five unrelated logs. The collector, dashboard, and alert channel are secondary. The join key is the product.

Option Incident reconstruction Operational cost Best fit
One error event with request context Fast for a single failing request Low Small services and early experiments
Error event plus trace and metric signals Strong across retries, dependencies, and cohorts Medium Marketplace rollouts with multiple services
Logs only Possible, but depends on exact field consistency Low at first, high during an incident Temporary diagnostics

My default is the middle row. Keep the error event useful on its own, then add standard trace context and low-cardinality metrics. That gives an operator a narrow path from a tenant-level failure to the request span, without turning every arbitrary tenant ID into a time-series label.

Start with the join key.

What does a marketplace API error need to prove?

An incident is a sequence, not a row in an error table. A buyer request enters the marketplace API, receives an experiment assignment, calls inventory or checkout, retries or fails, and may succeed after the assignment changes. The reconstruction needs those transitions in order.

The event should contain stable dimensions: service name, route template, deployment revision, experiment name, variant, tenant cohort, normalized error type, request ID, timestamps, duration, HTTP status, and a redacted dependency name. Do not put raw email addresses, authorization headers, payment details, or full request bodies into an error payload. Those fields make debugging riskier without making causality clearer.

There is a useful distinction here. tenantId is valuable for inspecting one incident, but it is usually a poor metric label because a marketplace can have many tenants. Record it on the event or trace. Aggregate metrics by cohort, experiment, route, and error class instead. OpenTelemetry describes metrics as measurements that are aggregated over time; that aggregation is exactly why high-cardinality identifiers belong elsewhere. See the metrics signal model in the References.

The same rule applies to error messages. “Inventory service timed out after 2000ms” is readable, but a changing item ID inside the message creates unstable groups. Store a normalized errorType such as DependencyTimeout and retain the short message as bounded detail. Grouping should be deterministic enough that the same failure does not become a new incident every time a request parameter changes.

How do you reconstruct the failure without guessing?

Start with one event schema. The schema below is deliberately boring. Boring schemas survive incident pressure.

Keep it boring.

type Cohort = "new-tenants" | "existing-tenants" | "high-volume"; type ApiErrorEvent = { occurredAt: string; service: "marketplace-api"; route: string; requestId: string; traceId?: string; revision: string; experiment: string; variant: string; cohort: Cohort; errorType: "DependencyTimeout" | "UpstreamRejected" | "Unexpected"; status: number; durationMs: number; dependency?: string; recoveredAt?: string; };

The variant and cohort values are not interchangeable. A variant describes the treatment. A cohort describes the population receiving it. During a rollout, the same variant may behave differently for new tenants because their catalog state or request volume differs. Keeping both fields is what lets an operator separate treatment damage from population-specific behavior.

In Node.js, the error boundary should enrich the event once, at the point where the request still has its context. It should then hand the record to a generic transport. The transport can write to a self-hosted collector, a queue, or an internal endpoint; the application should not know which one.

type EventSink = (event: ApiErrorEvent) => Promise<void>; type RequestContext = { requestId: string; traceId?: string; revision: string; experiment: string; variant: string; cohort: Cohort; }; function classify(error: unknown): ApiErrorEvent["errorType"] { if (error instanceof Error && error.name === "TimeoutError") { return "DependencyTimeout"; } if (error instanceof Error && error.name === "UpstreamRejectedError") { return "UpstreamRejected"; } return "Unexpected"; } async function captureApiError( error: unknown, context: RequestContext, sink: EventSink, startedAt: number, ): Promise<void> { const event: ApiErrorEvent = { occurredAt: new Date().toISOString(), service: "marketplace-api", route: "/v1/errors/events/{error_group_id}", requestId: context.requestId, traceId: context.traceId, revision: context.revision, experiment: context.experiment, variant: context.variant, cohort: context.cohort, errorType: classify(error), status: 502, durationMs: Date.now() - startedAt, dependency: "inventory", }; await sink(event); }

The example makes one opinionated choice: route is a template, not a literal path. That keeps groups and metrics stable. It also makes the status code explicit rather than deriving an error category from a free-form message. In a real service, the boundary should map known failures to their actual response status instead of treating every exception as a bad gateway.

Do not block the customer response on a remote error collector. Put a bounded event on a local queue, apply a timeout, and count dropped telemetry separately. Telemetry loss is a visibility problem; it should not become a marketplace outage. The queue needs a size limit, a redaction step, and a shutdown policy so a process cannot retain unbounded failed requests.

Which signals make the cohort comparison trustworthy?

One error event tells you that a request failed. It does not tell you whether the experiment caused the change. For that, emit a counter and a latency measurement with dimensions that can be aggregated: experiment, variant, cohort, route, status class, and normalized error type. Keep tenant identity in the event and trace, where it can be inspected for a bounded sample.

The useful comparison is not “errors increased.” It is closer to this:

type CohortWindow = { experiment: string; variant: string; cohort: Cohort; route: string; requests: number; failedRequests: number; dependencyTimeouts: number; p95DurationMs: number; }; function failureRate(window: CohortWindow): number { return window.requests === 0 ? 0 : window.failedRequests / window.requests; } function shouldPause(candidate: CohortWindow, baseline: CohortWindow): boolean { const candidateRate = failureRate(candidate); const baselineRate = failureRate(baseline); const relativeIncrease = baselineRate === 0 ? candidateRate > 0 : (candidateRate - baselineRate) / baselineRate; return relativeIncrease >= 0.25 || candidate.dependencyTimeouts > baseline.dependencyTimeouts * 2; }

Those thresholds are decision examples, not universal safety limits. Choose them from the experiment's traffic and harm model, and record the decision in the rollout policy. A 25% relative increase at two errors is not the same operational event as a 25% increase across 100,000 requests. Absolute counts, exposure, and time windows must travel with the comparison.

For reconstruction, preserve the trace identifier through the API boundary. W3C Trace Context defines the traceparent header format used to propagate trace identity between services. If a dependency call starts a new trace accidentally, the error record can still be found, but the causal chain is broken. That is the sort of small integration defect that costs more than adding another dashboard.

I also want a recovery signal. When the same request path succeeds after a failed attempt, associate the recovery with the request or trace where that is safe, and keep a separate success metric for the cohort. A retry that hides a dependency timeout is not the same as a clean request. Track both.

Where do common tracking choices stop helping?

Hosted error trackers and self-managed telemetry systems expose different boundaries. Sentry documents event capture, issue grouping, and tracing as separate concepts; that is useful when a team needs rich exception inspection but still needs a metric design for cohort rates. Rollbar documents occurrence grouping around item fingerprints and payload metadata; teams must choose those fields carefully or grouping will reflect changing request data. Bugsnag exposes error events and stability-oriented reporting, while cohort comparison still depends on the dimensions the application sends.

None of those boundaries removes the application design problem. A collector cannot infer that new-tenants is a population cohort, that variant-b is a treatment, or that two timeout messages represent one dependency failure. A platform can store, group, and query evidence. It cannot manufacture the missing causal fields.

That is why I measure time-to-first-call for the instrumentation itself. The first useful test should be a failed request in a local environment: does it produce one bounded event, with a stable group key, a trace identifier when available, and no credential material? If the answer requires a large configuration file or a custom adapter for every route, the design has already lost. Config bloat is an operational bug wearing a setup costume.

What should the resolution loop record?

Resolve the incident as a small experiment in reverse. First, identify the affected route, revision, variant, and cohort. Next, inspect representative traces, not just the aggregate rate. Then compare dependency timeout counts and duration against the control cohort for the same time window. Finally, record the intervention: pause exposure, roll back the revision, repair the dependency, or accept the signal as expected behavior.

The system should make that decision auditable. Store the event schema version, deployment revision, and experiment assignment at capture time. Keep raw payload retention short and aggregate metrics longer where policy permits. Redaction belongs before persistence, not in a dashboard query.

The runner-up approach is logs only. It can be the right choice for a tiny internal service, especially when a structured logger already carries request and trace fields. Its limitation is cross-cohort analysis: every consumer must agree on parsing, aggregation, and time windows. It becomes a poor fit when the marketplace needs a defensible treatment comparison. The more services join the incident, the more valuable a typed event and standard trace context become.

There is a second tradeoff. A hosted error tracker can reduce the time needed to inspect one exception, but it adds a service boundary, retention rules, and another event schema to govern. A self-managed collector gives more control over residency and routing, but it requires ownership of upgrades, storage, and on-call behavior. Neither option is universally better; pick hosted inspection for a small team with low integration surface, and self-managed telemetry when control and cross-service consistency outweigh setup time.

The final test is practical: can an engineer start from a cohort-level spike, open one failed request, follow its dependency edge, and see whether the same tenant later recovered? If not, add the missing field or propagation step before adding another alert. The useful tracking system is the one that shortens the path from a rate change to a defensible decision.

References