Short answer: put the HTTPS webhook endpoint in front of a durable queue, acknowledge only after the enqueue is committed, and let a rate-limited Node.js consumer own the business retry, nack, and dead-letter decisions.

That split is the useful architecture for payment and ledger systems because delivery speed and financial correctness have different failure boundaries. A sender needs a quick, unambiguous response; the ledger needs an idempotent effect and an audit trail that can survive a process crash. Treating one HTTP status as proof that both happened is how reconciliation queues become mysteries.

What does a webhook queue subscribe contract promise?

Start with a written contract, before selecting a queue library. Define which HTTPS status is an ack, what constitutes a nack, how long a sender may retry, and what identity makes two deliveries the same. “Push accepted” should mean that the receiver has authenticated the envelope and durably recorded it. It should not mean that a charge, transfer, or journal posting has completed.

The endpoint can be a small Node.js adapter, but the state it writes deserves domain names. An inbox record normally includes a subscription identifier, a delivery key, a hash or protected payload reference, the received timestamp, and a state such as ready, leased, retryable, applied, or dead. A unique constraint on (subscription_id, delivery_key) makes duplicate insertion an atomic fact rather than a best-effort cache lookup.

Ack after that commit. If the database is unavailable, return a retryable response and let the sender push again. If the same key arrives later, acknowledge it because its evidence already exists; do not run the financial side effect a second time. This is exactly-once thinking applied to the effect, not a claim that an HTTPS network can deliver exactly once.

That is the boundary.

Authentication and privacy sit on the same boundary. Verify the signature over the raw body, check freshness and replay windows, and keep account numbers out of ordinary logs. A dead-letter record should preserve enough information for an authorized reviewer to decide, while retention, encryption, deletion exceptions, and access review remain compliance controls rather than incidental queue settings.

How should a Node.js consumer combine rate limiting, ack, nack, and dead letters?

The consumer is a state machine with a capacity gate. It claims a ready row only when the limiter grants a token, leases it for a bounded interval, and records the downstream result. A dependency response that may succeed later moves the row to retryable with a next-attempt time. A malformed envelope or a policy rejection moves it to dead. A successful, idempotent business transaction moves it to applied.

Backpressure is policy — do not hide it inside a generic retry loop.

Choice Ack boundary Limiter behavior Cost or risk
Inline HTTPS processing After the side effect Request waits for capacity Few components, but sender timeout and downstream quota become one failure domain
Durable inbox plus worker After local enqueue Worker claims only admitted work Replay evidence and storage overhead
Database queue polling After worker outcome FOR UPDATE SKIP LOCKED spreads claims Local control with polling load
Scheduled batches At a batch checkpoint Batch size caps demand Predictable load, coarser latency

PostgreSQL describes FOR UPDATE SKIP LOCKED as useful for queue-like consumers that want to avoid lock contention. The skipped view is intentionally not a consistent report, so it is suitable for claiming work, not for reconciliation totals. A cron job can sweep expired leases or wake bounded work; cron is a scheduler, not an acknowledgement protocol.

Here is the critical path in Go. The HTTP handler is deliberately thin; a Node.js service can implement the same adapter while the transition rules remain explicit and testable.

package delivery import ( "context" "net/http" "time" ) type Inbox interface { PutIfAbsent(context.Context, string, string, []byte) (bool, error) ClaimReady(context.Context) (Message, bool, error) Retry(context.Context, string, time.Time, string) error Apply(context.Context, string) error DeadLetter(context.Context, string, string) error } type Message struct { Subscription string Key string Body []byte } type Limiter interface { Allow(time.Time) bool } func PushEndpoint(inbox Inbox) http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { // The adapter verifies the signature and reads the raw body first. key := r.Header.Get("X-Delivery-Key") subscription := r.Header.Get("X-Subscription") body := []byte{} // replace with the validated request body _, err := inbox.PutIfAbsent(r.Context(), subscription, key, body) if err != nil { w.WriteHeader(http.StatusServiceUnavailable) return } w.WriteHeader(http.StatusNoContent) } } func RunOne(ctx context.Context, inbox Inbox, limiter Limiter, apply func(context.Context, Message) error) error { if !limiter.Allow(time.Now()) { return nil // leave ready work for a later admission decision } msg, ok, err := inbox.ClaimReady(ctx) if err != nil || !ok { return err } if err := apply(ctx, msg); err != nil { return inbox.Retry(ctx, msg.Key, time.Now().Add(time.Second), err.Error()) } return inbox.Apply(ctx, msg.Key) } 

In production, body is the authenticated raw payload, not an empty slice. The example leaves parsing outside the transition code so the important question stays visible: what durable fact permits the ack? The apply function must use a business idempotency key as well; transport deduplication and ledger deduplication have different scopes.

Which failure signals keep the audit trail honest?

Record the delivery key, subscription, limiter decision, lease owner, downstream status, next-attempt time, and transition reason together. Useful alerts include oldest-ready age, lease age, retry volume by reason, dead-letter growth, and the gap between accepted deliveries and applied effects. A dashboard that counts only successful HTTPS responses can stay green while balances wait.

I once saw a limiter configured for 24 requests per second while another deployment emitted a startup burst. The downstream returned HTTP 429, but a retry helper counted the attempted call as a completed handler cycle. The queue accepted every delivery; 317 ledger updates quietly remained retryable until reconciliation exposed the drift. The first dashboard we checked showed healthy HTTP response counts, and the second showed workers busy, so the evidence was misleading in two different ways. We eventually traced the missing state through delivery keys, lease owners, and dependency response codes, then replayed a sample of the affected rows against an idempotent ledger write to prove that no balance had moved twice. The fix was not another retry. We made admission, the 429, and the next attempt a single recorded transition, then stopped claiming new rows while capacity was closed and added an alert on oldest-ready age.

Small records beat clever logs. Ack is a commit. If a worker dies after claiming a row, lease expiry makes the key eligible again; the idempotency constraint prevents a second posting. If the worker commits the posting and dies before marking applied, a reconciliation query should find the business key and finish the state transition without charging twice.

For an external side effect, persist an outbox intent and reconcile its result. Two databases do not share a transaction because a function awaits both calls. Deployment should add new states before workers emit them, keep older workers tolerant of expanded records, and canary with a lower admission rate so a schema or limiter change cannot flood the dependency.

When is an inbox the wrong choice?

The durable inbox costs storage, retention management, replay authorization, and an operator-visible backlog. The trade-off is worthwhile only when that evidence changes an operational decision. It is not suitable when the operation is tightly bounded, the sender's retry window comfortably exceeds that bound, the side effect is local and idempotent, and policy forbids retaining even a protected payload reference. In that case, direct processing can be correct, provided the timeout budget and duplicate-effect proof are documented.

Database polling fits a producer you control and a team that wants capacity decisions near its data. A scheduled batch fits coarse latency and a known reconciliation cutoff. Your mileage may vary: a small service may accept fewer moving parts, while a regulated ledger team may value durable arrival evidence more than request-level simplicity.

I'm not sure why inline rate-limit loops survive design reviews, except that one request is easy to demo and a backlog is not. Holding an HTTPS request while waiting for a token binds delivery deadlines, server concurrency, and dependency capacity into one timing equation. That is a poor place to hide a financial invariant.

The acceptance test is plain: stop a worker after claim, redeliver the same key, exhaust the limiter, send an invalid envelope, and replay a dead-letter record through an authorized review path. Balances must remain unchanged by duplicates, and every decision must be reconstructable from committed audit records.

References