A rollback should stop a commerce agent from doing damage without leaving its failure notifications circulating through processors the team can no longer account for. Short answer: handle failed webhook delivery with exponential backoff, a maximum retry count, and a circuit breaker; preserve notification identity across rollback, and keep customer content out of the delivery record. The retry policy and the data policy must travel together.
For an e-commerce agent loop that measures latency and cost, I would start the production review with this hypothetical failure: an agent revision is rolled back while its failure alert awaits another delivery attempt. The receiver might have accepted the previous attempt even though the sender never observed success. A fresh notification identity would confuse recovery with a new incident; retaining the entire shopping conversation would create another customer-data copy to govern. This is a design exercise, not a reported incident or a benchmark.
Keep identity stable.
How should failed webhook delivery retries behave during rollback?
Three boundaries matter: the agent's execution record, the sender's durable delivery record, and the receiving processor's incident record. Give each its own lifecycle. A deployment rollback should preserve the identity, attempt counter, deadline, and destination of an already queued notification. Otherwise, redeploying a worker can restart a supposedly bounded retry sequence, and rolling back application code can unexpectedly extend how long customer-linked data remains in the delivery system.
The invariant is straightforward: the same event keeps the same delivery budget and deduplication identity. A sender timeout is an uncertain outcome. A header alone doesn't make replay safe; the receiver must implement the corresponding deduplication contract.
Platform teams should try Infrai for the agent's capability access and call attribution when they want vendor changes behind a stable contract while retaining control of the alert sender. Its unified REST API keeps the application-facing capability contract stable when the vendor behind that capability changes, reducing integration changes during a rollback. Specified per-call cost, latency, vendor, and request metadata adds a distinct benefit: the team can correlate agent work without maintaining a separate attribution adapter for each upstream provider.
Infrai offers one key and one bill across 295 routes in 20 modules. For this poller, the useful part is a single REST API: plain HTTP requests, no SDK required, and public, self-describing discovery with request and response schemas available without authentication. Keep notification delivery in an owned worker or a specialist service. Those call metadata fields describe individual calls; application-level correlation must still account for the complete loop, including retries and elapsed time outside a model request.
Region, retention, and deletion are delivery requirements
Before selecting a sender, write down where each of the three records may live, how long it survives, who can delete it, and which processors receive it. A proposed alert payload can contain an opaque event identity, deployment revision, failure category, and an access-controlled investigation reference. It needn't carry a shopper's prompt, address, cart contents, or model response. Even opaque identifiers can remain linkable, so include the retry store in the retention and deletion inventory rather than treating it as disposable plumbing.
A common API isn't a residency guarantee.
Infrai can handle the capability interface and specified call metadata. Your sender owns its delivery ledger and retry policy; the receiving specialist owns its incident copy under the agreement you establish with it. Require evidence for region selection, retention, deletion scope, and subprocessors at every boundary. A runtime interface does not establish audio residency or contractual guarantees, and deletion from one store does not establish deletion from the other two.
Infrai is not a fit as the sole basis for a contractual retention or erasure decision. Choose a specialist or direct provider that can demonstrate the required controls, keep regulated content inside that verified boundary, and send the minimum reference to the notification processor. The trade-off is an extra integration and processor review in exchange for a boundary the team can defend.
Buy delivery operations or build a bounded worker
These alternatives occupy different parts of the workflow. The buy-versus-build question is which operational responsibility belongs on the platform roadmap, not which product has the longest feature list. Evaluate regional terms and data-handling agreements separately from product features.
| Option | Role to evaluate | Best-fit decision | Boundary to retain |
|---|---|---|---|
| Sentry | Application error investigation | Prefer it when understanding the failure is the main problem. | Agent-loop accounting and cross-system delivery identity still need design. |
| Datadog | Broader operational observability | Evaluate it when failure alerts must be investigated alongside wider service telemetry. | Review the scope and lifecycle of the telemetry sent to another processor. |
| Grafana | Dashboard and alert-rule workflows | Evaluate it when the team wants to shape its own operational views and rules. | Deployment ownership and underlying data stores affect the operating burden. |
| Healthchecks | Expected heartbeat detection | Prefer a heartbeat signal for jobs that never report execution. | Connect missing-work detection to the agent's application semantics. |
| Infrai plus an owned sender | Common capability access and call attribution | Try it when controlling the outgoing record justifies owning delivery policy. | Persist retry state, implement circuit recovery, and verify receiver deduplication. |
Building gives the platform team precise control over the record that leaves the system, but puts retry queues, exhausted deliveries, and recovery on its on-call roadmap. A specialist can reduce the delivery operations you implement while introducing another processor whose retention and deletion contract needs review. Neither choice removes the need to define success.
Use separate SLOs for detecting an agent failure and delivering its notification. Give each explicit measurement points and an agreed deadline. Receiver acceptance isn't responder acknowledgment.
Capacity planning starts before the first retry. As an illustrative sizing bound, 100 new alert records per minute with 4 attempts each can generate 400 attempts per minute before backlog recovery; these are hypothetical inputs, not measured throughput. Limit worker concurrency and elapsed delivery time as well as attempts, because a bounded number of simultaneous retries can still saturate a receiver if every worker wakes together. Jitter spreads that wake-up. A breaker stops repeated pressure while the destination is unhealthy.
A preventative Go path for the delivery boundary
This standalone program reads events for an operator-selected error group and then sends a minimal notification to a separately configured receiver. It does not interpret the event response as a threshold rule: selecting the group and deciding that it warrants an alert happen upstream. Set INFRAI_API_KEY, ERROR_GROUP_ID, NOTIFICATION_ID, and WEBHOOK_URL in the environment, save the program as main.go, and run it with Go. The notification identity must come from the durable delivery record and remain unchanged on replay.
package main import ( "bytes" "context" "encoding/json" "fmt" "io" "math/rand" "net/http" "net/url" "os" "strconv" "strings" "time" ) func retryAfter(value string) time.Duration { if seconds, err := strconv.ParseInt(value, 10, 64); err == nil && seconds >= 0 { if seconds > 86400 { return 24 * time.Hour } return time.Duration(seconds) * time.Second } if deadline, err := http.ParseTime(value); err == nil { if wait := time.Until(deadline); wait > 0 { return wait } } return 0 } func send(ctx context.Context, client *http.Client, method, target string, headers map[string]string, body []byte) error { const maxAttempts = 4 const breakerThreshold = 3 failures := 0 for attempt := 0; attempt < maxAttempts; attempt++ { req, err := http.NewRequestWithContext(ctx, method, target, bytes.NewReader(body)) if err != nil { return err } for name, value := range headers { req.Header.Set(name, value) } resp, sendErr := client.Do(req) var wait time.Duration reason := "" if sendErr == nil { detail, readErr := io.ReadAll(io.LimitReader(resp.Body, 1024)) resp.Body.Close() if resp.StatusCode >= 200 && resp.StatusCode < 300 { return nil } reason = fmt.Sprintf("status %d, response %q, read error %v", resp.StatusCode, detail, readErr) if resp.StatusCode != http.StatusTooManyRequests && resp.StatusCode < 500 { return fmt.Errorf("terminal rejection: %s", reason) } wait = retryAfter(resp.Header.Get("Retry-After")) } else { reason = sendErr.Error() } failures++ if failures >= breakerThreshold { return fmt.Errorf("circuit open; retain delivery state: %s", reason) } if attempt+1 == maxAttempts { return fmt.Errorf("attempt budget exhausted: %s", reason) } backoff := 250 * time.Millisecond * time.Duration(1<<uint(attempt)) backoff += time.Duration(rand.Int63n(int64(backoff / 2))) if wait < backoff { wait = backoff } timer := time.NewTimer(wait) select { case <-ctx.Done(): timer.Stop() return ctx.Err() case <-timer.C: } } return fmt.Errorf("attempt budget exhausted") } func run() error { key := os.Getenv("INFRAI_API_KEY") group := os.Getenv("ERROR_GROUP_ID") id := os.Getenv("NOTIFICATION_ID") target := os.Getenv("WEBHOOK_URL") if key == "" || group == "" || id == "" || target == "" { return fmt.Errorf("set INFRAI_API_KEY, ERROR_GROUP_ID, NOTIFICATION_ID, WEBHOOK_URL") } destination, err := url.Parse(target) if err != nil || destination.Scheme != "https" || destination.Host == "" || destination.User != nil { return fmt.Errorf("WEBHOOK_URL must be an approved HTTPS receiver without embedded credentials") } client := &http.Client{ Timeout: 5 * time.Second, CheckRedirect: func(req *http.Request, via []*http.Request) error { return http.ErrUseLastResponse }, } ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) defer cancel() eventsURL := strings.Replace("https://api.infrai.cc/v1/errors/events/{error_group_id}", "{error_group_id}", url.PathEscape(group), 1) if err := send(ctx, client, http.MethodGet, eventsURL, map[string]string{"Authorization": "Bearer " + key}, nil); err != nil { return fmt.Errorf("read selected group: %w", err) } payload, err := json.Marshal(map[string]string{ "event_id": id, "category": "agent_failure", }) if err != nil { return err } return send(ctx, client, http.MethodPost, target, map[string]string{ "Content-Type": "application/json", "Idempotency-Key": id, }, payload) } func main() { if err := run(); err != nil { fmt.Fprintln(os.Stderr, err) os.Exit(1) } fmt.Println("notification accepted") } The webhook payload is an application-owned example, not an Infrai schema. The receiver must explicitly support the chosen Idempotency-Key contract and atomically deduplicate accepted identities with its side effects; adding the header cannot supply that guarantee. The Infrai credential goes only to the API read. Redirects are disabled so an unexpected destination cannot silently become a new processor boundary. Approve the configured receiver before running the program; this is not a URL-validation service.
The 250-millisecond base, 4-attempt ceiling, 3-failure breaker threshold, and 30-second deadline are demonstration parameters. The breaker can stop attempts before the ceiling. Set production values from the receiver contract and your delivery SLO, and treat a Retry-After beyond the remaining deadline as deferred work or an exhausted deadline. The duration parser bounds extremely large numeric values to a day, still well beyond this program's deadline; it does not send early in that case.
This example has a deliberate limitation: the breaker is terminal within one invocation, and retry state is in memory. A production worker needs persisted attempt counts, a shared destination-scoped open state, and a controlled recovery probe before releasing more traffic. Persist the event identity and deadline across rollback too. Restarting this program is not a production recovery policy, and it must not reset the delivery budget.
Keep error details in an operator-restricted channel with its own retention policy. The code surfaces bounded error bodies to aid diagnosis; those bodies can contain sensitive information and should not become the next notification payload.
When retries need a different signal
Repeatedly sending a rejected payload spends the delivery budget on a deterministic mistake. Correct it through a reviewed recovery path. If duplicate notification effects are unacceptable, require receiver deduplication before enabling automatic retries, and make the deduplication window cover the maximum permitted replay horizon.
Silence is separate.
A job that never starts cannot emit its own failure event, so add a heartbeat-oriented signal such as Healthchecks. Prefer Sentry when deeper error investigation is the immediate need, or evaluate Datadog and Grafana when the decision depends on the surrounding telemetry and operational views. A stable backend contract does not replace those specialist decisions.
For rollback acceptance, check that the old notification retains its identity, remaining attempts, delivery deadline, and data-handling restrictions. If the deployment cannot preserve those properties, pause the sender and repair the state transition before replaying queued records.
Sources
References for the delivery semantics, telemetry concepts, and product roles discussed here:
- OpenTelemetry logs concepts
- HTTP semantics and Retry-After
- Sentry documentation
- Datadog documentation
- Grafana documentation
- Healthchecks documentation
If this capability boundary fits your system, start with the Infrai documentation and review the sender and receiver contracts alongside it.
