A Nodejs Express backend error tracking setup can capture an exception after an expensive model call, before a tool result is committed, or while the process is shutting down. Those cases look identical in an API dashboard if the system records only an exception string. The operational constraint changes the design: capture must preserve the agent loop step, attempt, latency, estimated cost, and commit state without duplicating an event during retries.
TL;DR: emit one normalized error envelope at the boundary where a loop step fails, give it a deterministic idempotency key, write it through a bounded asynchronous sink, and reserve process-level handlers for last-chance capture followed by termination. Measure signal quality by whether an event can answer three questions: which attempt failed, how much work preceded it, and whether any side effect committed. More volume is not better evidence.
How should Nodejs Express capture an error for tracking?
Start with an audit question rather than a dashboard question. A useful record has to distinguish a model timeout before a tool call from a ledger-write failure after the tool returned, because the safe retry decision is different. The minimum envelope therefore carries a trace identifier, loop and step identifiers, an attempt number, a stable operation name, elapsed time, a cost estimate in the smallest unit your accounting model supports, and an explicit commit state.
Keep the raw prompt and response out of the default event. This is partly signal hygiene and partly a compliance boundary: data minimization is easier to defend than an indefinite collection of payloads that might contain credentials, personal data, or payment data. PCI DSS defines scope around account data, while GDPR Article 5 includes data minimisation and storage limitation principles. Capture identifiers and classifications by default; attach content only through a separately governed diagnostic path.
The Go type below makes optionality visible. A missing estimate is not silently converted to zero, and an unknown commit state is not treated as uncommitted.
package errorsignal import "time" type CommitState string const ( CommitUnknown CommitState = "unknown" NotCommitted CommitState = "not_committed" Committed CommitState = "committed" ) type Event struct { EventID string `json:"event_id"` TraceID string `json:"trace_id"` LoopID string `json:"loop_id"` StepID string `json:"step_id"` Attempt int `json:"attempt"` Operation string `json:"operation"` ErrorClass string `json:"error_class"` Message string `json:"message"` Elapsed time.Duration `json:"elapsed"` EstimatedCost *int64 `json:"estimated_cost_minor,omitempty"` Commit CommitState `json:"commit_state"` OccurredAt time.Time `json:"occurred_at"` } Derive the event ID from stable execution coordinates instead of generating a random value on every send. The sink can then deduplicate a retry when a timeout leaves the sender uncertain about delivery. This does not create exactly-once execution; it creates an auditable approximation at the recording boundary. Keep that distinction explicit.
Duplicates are evidence too.
Capture at the step boundary first
Framework middleware sees request failures, but an agent loop usually contains several asynchronous operations inside one request or background job. Capture closest to the failed step, where attempt, timing, and commit state are still known, then let the error continue to the request or job boundary. A single recorder method also prevents each integration from inventing its own labels.
package errorsignal import ( "context" "crypto/sha256" "encoding/hex" "fmt" "time" ) type Sink interface { Record(context.Context, Event) error } type Step struct { TraceID, LoopID, StepID, Operation string Attempt int } func RunStep(ctx context.Context, sink Sink, s Step, fn func(context.Context) error) error { started := time.Now() err := fn(ctx) if err == nil { return nil } sum := sha256.Sum256([]byte(fmt.Sprintf("%s|%s|%d|%s", s.LoopID, s.StepID, s.Attempt, s.Operation))) e := Event{ EventID: hex.EncodeToString(sum[:]), TraceID: s.TraceID, LoopID: s.LoopID, StepID: s.StepID, Attempt: s.Attempt, Operation: s.Operation, ErrorClass: fmt.Sprintf("%T", err), Message: err.Error(), Elapsed: time.Since(started), Commit: CommitUnknown, OccurredAt: time.Now().UTC(), } if recordErr := sink.Record(ctx, e); recordErr != nil { return fmt.Errorf("step failed: %w; error signal failed: %v", err, recordErr) } return err } Do not swallow the original error after recording it. The caller still owns retry, rollback, or termination, and changing that control flow turns observability code into business logic. The compound error above also preserves a failed recording attempt instead of pretending the audit trail is complete.
This is where unhandledRejection and uncaughtException handling belong in a Node.js service conceptually: they are process boundaries, not ordinary recovery points. The Node.js process documentation warns that resuming normal operation after an uncaught exception is unsafe and describes the event as a last resort for synchronous cleanup before shutdown. Capture a minimal final event, stop accepting work, flush within a strict deadline, and terminate so a supervisor can replace the process. Do not turn either handler into a retry loop.
Make the sink bounded and idempotent
Error reporting can fail during the same network or storage disturbance that caused the application error. An unbounded queue converts that disturbance into memory pressure; a synchronous remote call adds failure latency to every request. Use a bounded channel, a short flush deadline, and an insert-if-absent contract keyed by event_id.
Keep it small.
package errorsignal import ( "context" "errors" ) var ErrQueueFull = errors.New("error signal queue full") type Writer interface { InsertIfAbsent(context.Context, Event) error } type BufferedSink struct { queue chan Event } func NewBufferedSink(size int) *BufferedSink { return &BufferedSink{queue: make(chan Event, size)} } func (s *BufferedSink) Record(_ context.Context, e Event) error { select { case s.queue <- e: return nil default: return ErrQueueFull } } func (s *BufferedSink) Run(ctx context.Context, w Writer) error { for { select { case <-ctx.Done(): return ctx.Err() case e := <-s.queue: if err := w.InsertIfAbsent(ctx, e); err != nil { return err } } } } A production worker needs an explicit policy when persistence fails: retry with a cap, spill to a governed local store, or report a counter and drop. There is no universal answer because durability, data sensitivity, and shutdown budgets differ. Count queue saturation and failed persistence separately from application exceptions, or a quiet dashboard may mean a broken reporter rather than a healthy service.
Consider one concrete retry sequence. Attempt 1 completes a model call, starts a tool write, and loses the connection before receiving an acknowledgement; the application knows the elapsed time and estimated model cost, but the write's commit state is unknown. It emits an event whose ID is derived from the loop, step, attempt, and operation. During shutdown the same envelope is sent again, and the receiver's insert-if-absent operation retains one copy. Attempt 2 must not begin merely because the dashboard accepted that event: an operator or reconciliation worker first resolves the unknown commit state against the system of record. This separation matters because deduplicating error records says nothing about deduplicating the underlying tool side effect. It also explains why one generic exception counter, despite looking tidy, is too weak for an agent loop that can spend money and mutate state before failure.
OpenTelemetry's logs model is useful because it treats logs, traces, and metrics as correlated signals, including trace and span context on log records. Preserve that correlation even if the first sink is a file or database table. The storage destination can change; the event contract should not.
Separate latency and cost signal from noise
An agent failure count cannot tell an operator whether the system lost one cheap validation step or repeated several high-latency model calls before failing. Aggregate by stable dimensions such as operation, error class, model family, and commit state, while keeping trace IDs for drill-down. Never use raw messages, prompts, user IDs, or step IDs as metric labels: their cardinality grows with traffic and their content may be sensitive.
For every completed attempt, successful or failed, record elapsed time and the input data used by the cost estimator. Keep the estimator version alongside the result. Cost rules change, and an unversioned estimate cannot be reconciled later. If usage data is absent, store unknown rather than zero. Zero means the operation was measured and consumed none; unknown means the evidence is incomplete.
Retain all terminal failures and all events with an unknown commit state; sample repetitive, pre-commit validation failures only after their aggregate counter is recorded. This favors evidence needed for safe replay over sheer dashboard volume. It also gives a concrete test for any hosted, self-managed, or hybrid backend: can it preserve deterministic event IDs, bounded attributes, retention controls, and trace correlation without requiring payload capture?
Price comes later. Compare total ingestion behavior, retention, exportability, redaction, regional controls, and the labor required to operate the pipeline. A low entry price cannot repair an event model that loses attempt or commit state.
Evidence first.
Roll out without corrupting the audit trail
Treat instrumentation as a behavior change because it adds allocation, queueing, and shutdown work. Begin in shadow mode: construct and validate envelopes while sending only aggregate health counters. Then enable persistence for a small cohort selected by a stable hash of loop ID, compare application latency and reporter saturation, and expand only when acceptance criteria hold. Martin Fowler's discussion of feature toggles explains why cohort-based release control and toggle retirement matter; leave an owner and removal date on the rollout switch.
Test four failure paths before increasing the cohort: the step function fails, the queue is full, the writer is unavailable, and shutdown begins with queued events. Verify that the original failure remains visible in every case, duplicate sends create one stored event, unknown cost remains unknown, and an unknown commit state blocks automated replay. Those are reconciliation tests, not dashboard screenshots.
The migration is compact: standardize the envelope, dual-write to the old and new sinks with the same event ID, reconcile counts and sampled records by cohort, switch reads, then retire the old path and its toggle. Keep the raw comparison long enough to explain discrepancies under the organization's retention policy. A trustworthy error dashboard is the consequence of this chain of evidence, not the starting point.
