SaaS Error Event Query API in Node.js (with Rollback-Safe Imports)

Use a queryable error event API plus a separate schedule monitor when a Node.js SaaS marketplace needs alerting for imports that stop producing results. The decision rule is strict: advance an import watermark only after the result is durable, and make alert delivery retryable without letting notification state rewrite that watermark. A thrown error is evidence of one failure; missing expected output is evidence that the workflow itself may have stopped.

TL;DR: emit structured start and terminal events from the Node.js worker, store them behind a small query API, and run an independent cron monitor that compares each schedule's deadline with its last successful result. Send Slack or email through an idempotent outbox. This design makes silence observable, preserves an audit trail, and lets a deployment roll back without erasing or duplicating the operational truth.

Should a Node.js SaaS poll an error alerting API?

A marketplace import has at least three clocks: the scheduled start, the worker's actual start, and the durable completion. Treating those as one timestamp creates an ambiguity that no alert transport can repair. A job may never be enqueued, may start and stall, or may commit its rows while the process dies before reporting success. Those failure boundaries require different evidence.

Silence is data.

The invariants are small enough to put in an architecture decision record:

1. Every scheduled occurrence has a stable run_id, derived before execution and reused by retries.
2. A terminal event is written only after the imported result is durable.
3. The success watermark moves monotonically; an older retry cannot move it backward.
4. Alert attempts have their own idempotency key and audit history.
5. The monitor is outside the worker's failure domain and evaluates both explicit failures and absent terminal events.

The exactly-once mindset belongs here, but “exactly once” should be expressed as enforceable state transitions, not as faith in a queue. Delivery can repeat. A unique key on (run_id, event_type) and a compare-and-swap watermark turn those repeats into stable facts, while the audit log retains rejected or duplicated attempts for reconciliation.

Retries happen.

There is also a compliance boundary. Import events should carry identifiers and operational counts, not raw seller records, payment details, credentials, or webhook payloads. Retention, access, and deletion periods depend on the marketplace's applicable policy and jurisdiction; the architecture should therefore support configured limits rather than claim one universal number.

Decision: ledger first, notifications second

The query API is the system of record for operational evidence. It can answer, for one schedule, which run was expected, what state transitions were accepted, when the last durable result appeared, and which alerts were attempted. Slack and email are downstream delivery channels. A lost message can be retried; a message must never manufacture job state. This architecture has a real limitation: it adds a ledger, an expectation model, a monitor, and an outbox that somebody must operate. It is unsuitable for an ad hoc task whose failure is already synchronously returned to a human and whose result has no reconciliation value; in that narrower case, direct exception notification is the better boundary. The extra machinery earns its place only when absence itself is actionable and rollback must preserve what operators know.

This ordering matters during rollback. Suppose release B writes a new optional event field and is rolled back to release A. If readers tolerate unknown fields and the core event envelope remains stable, A can continue from the existing watermark. The rollback neither deletes B's events nor interprets notification delivery as successful import execution. Schema evolution should be additive until every active reader can handle the new representation.

Polling is acceptable here because the question is bounded and replayable: “which expected runs lack a terminal result before their deadline?” The API should paginate on a stable cursor, return event time and ingestion time separately, and support lookup by schedule_id and run_id. Avoid exposing an unbounded free-form search interface to the monitor; a narrow contract is easier to index, authorize, test, and preserve across rollback.

The critical path in Go

The production worker may be Node.js, but the state machine is language-independent. The Go example below makes the transactional boundary explicit, keeps all code in one place, and avoids binding the design to an alerting vendor.

package imports import ( "context" "fmt" "time" ) type Store interface { CommitResultAndEvent(ctx context.Context, result Result, event Event) error AdvanceWatermark(ctx context.Context, scheduleID string, completedAt time.Time) error MissingTerminals(ctx context.Context, dueBefore time.Time) ([]ExpectedRun, error) InsertAlertOnce(ctx context.Context, alert Alert) (bool, error) } type Result struct{ RunID string } type Event struct { RunID, ScheduleID, Type string OccurredAt time.Time } type ExpectedRun struct { RunID, ScheduleID string DueAt time.Time } type Alert struct { Key, RunID, Channel string CreatedAt time.Time } func Complete(ctx context.Context, db Store, result Result, scheduleID string, now time.Time) error { event := Event{RunID: result.RunID, ScheduleID: scheduleID, Type: "import.completed", OccurredAt: now} if err := db.CommitResultAndEvent(ctx, result, event); err != nil { return fmt.Errorf("commit result and terminal event: %w", err) } // The store must reject a timestamp older than the current watermark. return db.AdvanceWatermark(ctx, scheduleID, now) } func DetectSilence(ctx context.Context, db Store, now time.Time) error { runs, err := db.MissingTerminals(ctx, now) if err != nil { return fmt.Errorf("query overdue runs: %w", err) } for _, run := range runs { alert := Alert{ Key: "missing-terminal:" + run.RunID, RunID: run.RunID, Channel: "operations", CreatedAt: now, } if _, err := db.InsertAlertOnce(ctx, alert); err != nil { return fmt.Errorf("enqueue alert %s: %w", run.RunID, err) } } return nil } 

CommitResultAndEvent must be atomic within the chosen persistence boundary. If the imported rows and event ledger cannot share a transaction, use a transactional outbox tied to the result commit, then publish from that outbox. Do not mark completion before the durable result exists.

The short function conceals an important distinction: InsertAlertOnce records intent, not successful delivery. A dispatcher can retry Slack or email and append attempt outcomes without re-alerting the same run under the same policy. If escalation rules intentionally create a second notification, they need a different key, such as a policy version and escalation stage. This makes acknowledgements and re-notifications explainable months later.

Operating the monitor without inventing certainty

Choose the silence deadline from the schedule contract: expected start, allowed lateness, and the longest accepted execution window. Do not derive it from an attractive round number. A daily catalog import and a five-minute inventory feed have different lateness semantics, while a maintenance pause should suppress expectations rather than generate a stream of false failures.

Three timestamps should survive the whole path: event occurrence, ingestion, and alert creation. OpenTelemetry's logs data model distinguishes observed time from event time and provides fields for trace and span context, which is useful when a scheduled run also emits traces. The correlation is evidence, however, not the durable job state; a sampled or unavailable trace must not make reconciliation impossible.

Test the absence path deliberately. Freeze the scheduler clock, create an expected run, omit its terminal event, and verify that one alert intent appears after the configured deadline. Then run the monitor twice. One intent should remain. Also test a late success, an older retry after a newer success, a dispatcher crash after sending but before recording the response, and a rollback in which the older reader encounters an event written by the newer release.

Late success needs an explicit policy. It may resolve an open incident automatically, append a recovery notification, or require an operator to reconcile partially imported data. Pick one rule per import class and record the transition. Silence is uncertain by definition; the system knows that evidence is missing, not that zero rows were committed.

Keep the monitor observable as well. Record its last completed evaluation, query cursor, number of schedules examined, and age of its oldest pending alert. A monitor that silently stops is merely a second scheduled import with the same original problem.

Rejected option and where it still fits

The rejected primary design is an error webhook emitted directly from the Node.js catch path. It is appealing because explicit exceptions reach a chat channel quickly, and it has fewer moving parts. It cannot, by itself, detect that cron failed to launch the process, that a worker remained alive without progress, or that a result committed before notification code ran. It also couples rollback safety to whichever payload shape the worker release emits.

Keep that webhook as a secondary fast path when low latency matters. Give it the same run_id, route it through the idempotent alert outbox, and reconcile it against the ledger so an explicit failure and the later silence scan do not open two incidents. This division is deliberate: the worker reports what it observed, while the monitor asks what should have happened but did not leave durable evidence. The trade-off is operational complexity in exchange for a recoverable record; teams that cannot own the monitor should retain the simpler callback boundary and accept that it cannot detect every silent run.

The selection criterion is therefore not the smallest alert API. It is the smallest architecture that can prove a scheduled occurrence, preserve a durable terminal fact, query missing facts independently, and replay notification delivery without corrupting those facts. For marketplace imports, rollback-safe watermarks and auditable alert intents carry more decision weight than transport convenience.

References

• https://opentelemetry.io/docs/concepts/signals/logs/