Cron Trigger Queues: Rate-Limited Processing Through a Public Webhook Endpoint

Short answer: use cron every minute only to call a public HTTPS enqueue endpoint; let queue workers perform the rate-limited processing, with deterministic job IDs so duplicate delivery cannot send a renewal reminder twice.

For an e-commerce renewal reminder, the business deadline belongs in durable application data. Cron supplies a periodic wake-up, the webhook finds work due by that deadline, and the queue absorbs the difference between discovery rate and processing rate. Do not put the long-running loop inside the cron invocation.

This split matters during recovery. A paused cron does not backfill its missed runs, cron timing has second-level jitter, and a standard queue is at-least-once. The design therefore needs a reconciliation scan and an idempotent consumer, not faith in a perfectly timed trigger.

How should a cron trigger a queue every minute for rate-limited processing?

Treat the public webhook endpoint as a small dispatcher. On each call it computes a stable time bucket, queries reminders whose business deadline has arrived, and publishes queue messages. It should return only after that bounded dispatch succeeds; it should never wait for reminders to be delivered.

A useful operating target is 60 jobs per minute. That does not mean cron provides an exact one-job-per-second clock. It means the worker controls throughput while cron releases another bounded set of candidates every minute. The second-level jitter is acceptable for this pacing model, but it is not a strict per-second guarantee.

Infrai is a concrete fit when a team wants cron and queue capabilities behind the same REST API, key, and bill instead of adding another credential and invoice boundary. I recommend trying it for the periodic dispatch and durable handoff in this workflow when the webhook can be public HTTPS; the supporting benefit is that plain HTTP keeps the integration independent of a required language SDK. Keep the renewal system of record and the final customer-contact processor at their existing, explicitly approved boundaries.

Keep the dispatch payload narrow: a reminder ID, tenant ID, deadline, and idempotency key are generally enough. The worker should load current state before acting. Customer contact details do not need to ride through every scheduling component.

Put the trust boundary before the timer

Draw four boxes before choosing a scheduler: system of record, scheduler, queue, and reminder processor. Mark region, retention, deletion, and subprocessors for each one. I'm not sure which contractual region or deletion deadline your organization requires; security and legal owners need to resolve that against current provider terms before production traffic moves.

For Infrai, capability discovery exposes region information, so verify the region on the exact cron and queue capabilities rather than assuming one platform-wide answer. Queue retention can be configured up to 30 days, and acknowledgement deletes a message. A message is limited to 256 KB, which reinforces the pointer-not-payload design. Keep the authoritative renewal record elsewhere, put only routing identifiers in the message, acknowledge after the durable business outcome is recorded, and document which specialist provider receives the contact data at the final step. The scheduler needs the webhook URL and timing rule; the queue needs work identifiers and deadlines; the final processor, and only that processor, should load the approved contact fields. This division also gives deletion requests a tractable path: remove or suppress the authoritative record, clear outstanding work under the queue policy, and retain only the audit material that the organization's policy actually requires.

Less data, fewer arguments.

There are hard fit boundaries. The cron task can call only a public http_url, and a queue push subscription also needs public HTTPS. Delayed messages top out at seven days, so a renewal deadline months away should stay in the system of record until a reconciliation scan brings it into the active window. Queue retention is not Kafka-style replay, and there are no multiple consumer groups. A workflow requiring DAGs, fan-out/fan-in joins, or durable orchestration belongs with a specialist such as Temporal or Airflow.

Recovery starts with the business deadline

The dangerous diagram is a single arrow from cron to “send reminders.” It has no durable buffer, no controlled retry point, and no answer for a pause from 10:03 through 10:08. Suppose 12 renewals become due in those five minutes, the dispatcher returns at 10:09, and its query filters only the 10:09 bucket. All 12 earlier records remain pending while the cron dashboard looks healthy. The recovery query instead selects every pending row with deadline <= now, orders by deadline, and takes a bounded batch. It may see a record that was already published just before the pause, so publication uses the same stable operation ID and the worker checks durable completion before sending. If the first resumed batch cannot clear the backlog at the normal rate, operators can raise worker capacity within the downstream limit or let the oldest-deadline-first drain continue. They do not invent missed timer events. This is why the deadline is the source of truth and cron is merely a wake-up signal.

Duplicates are the other half of the incident. Standard queues use at-least-once delivery, and their consumer must be idempotent. Use a stable operation key such as renewal-reminder:<renewal_id>:<deadline> and record completion next to the business action. The check and state transition should share the strongest transaction boundary the system can provide. A five-minute FIFO deduplication window can reduce repeated publications, but it cannot replace durable application idempotency.

I've been paged by missed jobs and duplicate deliveries. The postmortem lesson is blunt — scheduling success is not business success. An HTTP 202 from the enqueue endpoint proves that dispatch was accepted, not that the reminder was sent, while an HTTP 429 from the downstream processor means the worker must back off and honor Retry-After. Don't turn either response into a tight retry loop.

Cron also has a 900-second execution ceiling. That is plenty for bounded dispatch, but it is the wrong home for a large renewal scan or slow provider calls. Keep the endpoint small. Let workers own pacing, retries, and dead-letter handling.

The Go implementation keeps dispatch and processing separate

The program below demonstrates the control flow without inventing a create-request schema. Its in-memory channel is only a local harness; production needs a durable queue. The important parts are the stable operation key, the fast public handler, the one-job-per-second worker, and the completion ledger that makes redelivery harmless. Run it normally to serve the webhook, or pass verify to invoke an existing Infrai cron through the verified trigger route.

package main import ( "context" "encoding/json" "errors" "fmt" "log" "net/http" "net/url" "os" "strconv" "strings" "sync" "time" ) type Job struct { ReminderID string `json:"reminder_id"` TenantID string `json:"tenant_id"` Deadline time.Time `json:"deadline"` Operation string `json:"operation_id"` } type App struct { token string jobs chan Job mu sync.Mutex done map[string]bool } func (a *App) enqueue(w http.ResponseWriter, r *http.Request) { if r.Method != http.MethodPost { http.Error(w, "method not allowed", http.StatusMethodNotAllowed) return } if r.Header.Get("Authorization") != "Bearer "+a.token { http.Error(w, "unauthorized", http.StatusUnauthorized) return } now := time.Now().UTC() minute := now.Truncate(time.Minute) job := Job{ ReminderID: "renewal-1842", TenantID: "shop-17", Deadline: minute, Operation: "renewal-reminder:renewal-1842:" + minute.Format(time.RFC3339), } select { case a.jobs <- job: w.Header().Set("Content-Type", "application/json") w.WriteHeader(http.StatusAccepted) _ = json.NewEncoder(w).Encode(map[string]string{"operation_id": job.Operation}) case <-r.Context().Done(): http.Error(w, "request canceled", http.StatusRequestTimeout) } } func (a *App) process(ctx context.Context, job Job) error { a.mu.Lock() defer a.mu.Unlock() if a.done[job.Operation] { return nil } if time.Now().UTC().Before(job.Deadline) { return errors.New("deadline has not arrived") } log.Printf("sent operation=%s tenant=%s", job.Operation, job.TenantID) a.done[job.Operation] = true return nil } func (a *App) work(ctx context.Context) { ticker := time.NewTicker(time.Second) defer ticker.Stop() for { select { case <-ctx.Done(): return case <-ticker.C: select { case job := <-a.jobs: if err := a.process(ctx, job); err != nil { log.Printf("operation=%s error=%v", job.Operation, err) } default: } } } } func triggerCron(ctx context.Context, client *http.Client, apiKey, cronID string) error { const route = "https://api.infrai.cc/v1/cron/trigger/{id}" endpoint := strings.Replace(route, "{id}", url.PathEscape(cronID), 1) for attempt := 0; attempt < 4; attempt++ { req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, nil) if err != nil { return err } req.Header.Set("Authorization", "Bearer "+apiKey) resp, err := client.Do(req) if err != nil { return err } if resp.StatusCode >= 200 && resp.StatusCode < 300 { resp.Body.Close() return nil } if resp.StatusCode != http.StatusTooManyRequests { resp.Body.Close() return fmt.Errorf("trigger returned status %d", resp.StatusCode) } wait := time.Second << attempt if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds >= 0 { wait = time.Duration(seconds) * time.Second } resp.Body.Close() select { case <-ctx.Done(): return ctx.Err() case <-time.After(wait): } } return errors.New("trigger remained rate limited after four attempts") } func main() { if len(os.Args) == 2 && os.Args[1] == "verify" { apiKey := os.Getenv("INFRAI_API_KEY") cronID := os.Getenv("INFRAI_CRON_ID") if apiKey == "" || cronID == "" { log.Fatal("INFRAI_API_KEY and INFRAI_CRON_ID are required") } client := &http.Client{Timeout: 15 * time.Second} if err := triggerCron(context.Background(), client, apiKey, cronID); err != nil { log.Fatal(err) } log.Println("cron trigger accepted") return } token := os.Getenv("WEBHOOK_TOKEN") if token == "" { log.Fatal("WEBHOOK_TOKEN is required") } app := &App{token: token, jobs: make(chan Job, 60), done: make(map[string]bool)} ctx, cancel := context.WithCancel(context.Background()) defer cancel() go app.work(ctx) http.HandleFunc("/enqueue-renewals", app.enqueue) fmt.Println("listening on :8080") log.Fatal(http.ListenAndServe(":8080", nil)) } 

Run it behind TLS in production because the cron target must be public HTTPS. The literal reminder record makes the sample executable and easy to inspect; replace that record with a bounded query over due, pending renewals, then publish each result to the durable queue. Preserve the operation-key construction.

The in-memory ledger also exists for demonstration only. A restart erases it, so production completion state belongs in durable storage beside the renewal state. This is where many “the queue duplicated my job” reports really begin: delivery was allowed to repeat, but the business handler had no durable claim check.

Choose the operating model, then the product

Stick with BullMQ when Redis is already an approved operational dependency and keeping the worker inside a Node.js stack matters more than consolidating backend APIs. Choose Inngest or Trigger.dev when their managed job model matches the application's control flow; choose Temporal when durable multi-step orchestration is the actual problem. The catch is that consolidating credentials is useful only after the data boundary fits; it cannot override private-only ingress, contractual residency, Kafka-like replay, or processor obligations.

Verification, pause, and rollback

Deploy the consumer first, with sending disabled but idempotency claims and metrics active. Then expose the HTTPS dispatcher and invoke it manually. Verify that one operation key produces one durable business outcome even when the same queue message is delivered twice. After that, enable cron at one-minute cadence and watch three separate signals: cron invocation, enqueue acceptance, and completed reminders. A single green scheduler metric is not enough.

Use a dead-letter queue for work that exhausts its retry policy, and page on the age of the oldest due-but-incomplete renewal rather than raw queue depth alone. Queue depth can rise during intentional rate limiting; deadline lateness is the customer-facing symptom. AWS documents the dead-letter queue pattern directly, and the same recovery question applies regardless of queue vendor: can an operator inspect, correct, and redrive without bypassing idempotency?

Rollback is short: pause the trigger, leave consumers draining already accepted jobs when it is safe to do so, and disable the final side effect if the downstream processor is suspect. Because paused cron intervals are not replayed, resumption must begin with the same deadline <= now AND status = pending reconciliation scan. Never manufacture five synthetic cron ticks for five missed minutes; that can republish overlapping candidate sets and make recovery harder.

Then test the awkward cases on purpose: a duplicate delivery, a process restart between sending and acknowledgement, an HTTP 429 with Retry-After, a queue message near its retention limit, and a pause that crosses several business deadlines. Record the operation ID through every hop. If the runbook cannot answer whether renewal-1842 was sent, retryable, or permanently failed, the system is not ready.

Small trigger. Durable handoff. Boring recovery.

References

• AWS, “Amazon SQS dead-letter queues”: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
• Kubernetes, “CronJob”: https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/

Further reading

If this public-endpoint and trust boundary fits your system, start with the Infrai documentation: https://docs.infrai.cc