Building an In-Memory Circuit Breaker for Webhook Ingress: Shedding Load Without Losing Events

Webhooks connect cloud systems. Payment providers like Stripe, code hosts like GitHub and commerce platforms like Shopify all use them to tell your services that something changed. When a sender has a bad day, you pay for it. A bulk export, a backfill or a burst of retries can hit your ingress endpoint, and the endpoint is the first thing that fails.

When a burst arrives, worker threads or goroutines block on slow downstream calls and connection pools saturate. Memory fills with queued request bodies and garbage-collection pressure pushes latency higher. Health checks start failing, the load balancer moves traffic to the remaining nodes, and those nodes tip over too.

This guide describes an in-memory admission controller for webhook ingress. It combines an adaptive concurrency limit, a CoDel-style delay signal and priority-aware shedding. It also covers what “not losing events” means when you reject a request, because that depends on how each sender retries.

1. How an ingress meltdown unfolds

Webhook senders enforce short response deadlines, and each provider sets its own:

Sender	Documented behavior
GitHub	Expects a 2XX within 10 seconds, otherwise it terminates the connection and counts a failed delivery. It does not redeliver automatically; failed deliveries must be redelivered manually or via the API.
Shopify	One-second connection timeout and five-second timeout for the whole request. Failed deliveries are retried 8 times over 4 hours.
Slack (Events API)	Expects a valid response within three seconds.
Stripe (live mode)	Retries for up to 3 days with exponential backoff.

Because the deadlines are short, a slow endpoint produces timeouts, and timeouts produce more traffic. A sender that retries adds those retries to the stream of fresh events. A sender that doesn’t retry, like GitHub, leaves you with lost events and manual redelivery.

The failure then plays out in a predictable order:

Connection saturation: new connections queue in the socket backlog.
Worker starvation: handlers blocked on database calls hold their threads or goroutines.
Memory growth: request bodies pile up while waiting for a worker.
Latency amplification: garbage-collection work and contention raise latency, which causes more sender timeouts.

[Upstream Sender] ──(flood / retries)──> [Load Balancer] │ ▼ [Ingress Worker Node] ├── Connection queue (full) ├── Workers (blocked) └── Memory / GC (pressure) ──> [meltdown]

The first defense is architectural. GitHub’s documentation recommends putting a queue in front of slow work so the server can respond immediately and process the payload in the background. Shopify gives the same advice for bursts. Admission control is the second line of defense. It protects the thin “validate, enqueue, acknowledge” path itself.

2. Why the admission decision should be local

Resilience4j’s circuit breaker is the standard tool for outbound calls. It is an in-process library and doesn’t need Redis or any other shared store. The same reasoning applies at ingress:

No extra dependency on the hot path. Consulting a shared store for every inbound request adds a network round trip and a new failure mode. If that store is down, your ingress should still be able to shed load.
The overload is local. Thread pools, memory and database connections are per-node resources. A node-local signal reflects what that node can actually handle.
Senders only see HTTP. They react to status codes and timing, so the cleanest signal you can give them is a fast, explicit rejection.

The trade-off is that each node decides independently. With uneven load balancing, one node may shed while another has spare capacity. Per-node metrics (section 7) show you when that happens.

3. Two mechanisms: adaptive concurrency and queue delay

Adaptive concurrency limits

Little’s Law says that concurrency equals throughput times latency. A fixed limit such as “100 concurrent requests” goes stale as soon as latency changes. If the database slows down, 100 is too high. If it speeds up, 100 is too low.

Netflix’s open-source concurrency-limits library, described in their 2018 post “Performance Under Load”, adjusts the limit from observed latency instead. Its gradient algorithm uses:

newLimit = gradient × currentLimit + queueSize

In the library’s Gradient2 variant, the gradient is the long-term RTT divided by the current RTT, clamped to the range 0.5 to 1.0. The result is then smoothed (default factor 0.2):

gradient = max(0.5, min(1.0, longTermRTT / currentRTT)) newLimit = gradient × currentLimit + queueSize newLimit = currentLimit × (1 − smoothing) + newLimit × smoothing

When latency rises above its long-term average, the gradient drops below 1 and the limit shrinks. When latency is stable, the queueSize term lets the limit creep upward. It can be a fixed value or a function of the limit, such as a square root.

Queue delay (CoDel-style)

Counting 5xx errors reacts late, because by the time errors appear the damage is done. A better early signal is how long requests wait for a slot.

CoDel (Controlled Delay) was designed by Kathleen Nichols and Van Jacobson as an active queue management algorithm for network buffers (RFC 8289). It tracks a packet’s sojourn time, the time spent in the queue. It acts only when the minimum delay over a sliding window stays above a target, which allows short bursts but prevents a persistent standing queue. RFC 8289’s defaults are a 5 ms target and a 100 ms interval.

Facebook adapted a variant of CoDel for request queues, as described in “Fail at Scale”. Its open-source Wangle library shows the idea. Within each interval it tracks the minimum queueing delay and declares overload if that minimum exceeded the target. While overloaded, it sheds requests that have waited longer than twice the target. This is an adaptation rather than textbook CoDel. Treat 5 ms and 100 ms as starting points and tune them against your handler’s normal service time.

4. Architecture

Requests pass through classification and then a limiter that both enforces the adaptive limit and measures queue delay:

Incoming webhook request │ ▼ ┌──────────────────────────────────────────┐ │ 1. Priority classifier │ │ (route or verified partner identity) │ └──────────────────────────────────────────┘ │ ▼ ┌──────────────────────────────────────────┐ │ 2. Admission: slot free? │ │ yes → admit; no → queue briefly, or │ │ shed if overloaded (by priority) │ └──────────────────────────────────────────┘ │ (slot acquired) ▼ ┌──────────────────────────────────────────┐ │ 3. Thin handler: verify signature, │ │ enqueue durably, return 2xx │ └──────────────────────────────────────────┘ │ ▼ Release slot → feed RTT back to the limit

Classify by something you control, such as the endpoint path or the partner identified after signature verification. Don’t trust a priority header that arrives from the public internet, because anyone can set it.

5. A working Go sketch

The following sketch uses only the standard library. It combines a Gradient2-style limit with a Wangle-style overload flag. The parameter values are illustrative and need tuning against your own traffic.

package ingress import ( "math" "net/http" "sync" "time" ) type Priority int const ( PriorityLow Priority = iota PriorityDefault PriorityHigh ) type Config struct { InitialLimit float64 // e.g. 20 MinLimit float64 // e.g. 4 MaxLimit float64 // e.g. 200 Smoothing float64 // e.g. 0.2 TargetDelay time.Duration // e.g. 5ms Interval time.Duration // e.g. 100ms MaxWait time.Duration // longest a request may wait for a slot, e.g. 2 * TargetDelay MaxQueue int // hard cap on waiting requests } type waiter struct { ready chan struct{} granted bool } type Limiter struct { cfg Config mu sync.Mutex limit float64 inflight int waiters []*waiter shortRTT float64 // seconds, fast EWMA longRTT float64 // seconds, slow EWMA intervalEnd time.Time minDelay time.Duration overloaded bool } func NewLimiter(cfg Config) *Limiter { return &Limiter{cfg: cfg, limit: cfg.InitialLimit} } // Acquire returns a release func and true if the request is admitted. func (l *Limiter) Acquire(p Priority) (func(time.Duration), bool) { l.mu.Lock() if l.inflight < int(l.limit) && len(l.waiters) == 0 { l.inflight++ l.observeDelay(0) l.mu.Unlock() return l.release, true } if l.shouldShed(p) { l.mu.Unlock() return nil, false } w := &waiter{ready: make(chan struct{})} l.waiters = append(l.waiters, w) l.mu.Unlock() start := time.Now() timer := time.NewTimer(l.cfg.MaxWait) defer timer.Stop() timedOut := false select { case <-w.ready: case <-timer.C: timedOut = true } l.mu.Lock() defer l.mu.Unlock() l.observeDelay(time.Since(start)) if timedOut && !w.granted { l.removeWaiter(w) return nil, false } return l.release, true } func (l *Limiter) shouldShed(p Priority) bool { switch p { case PriorityHigh: return len(l.waiters) >= l.cfg.MaxQueue // only when the queue is full case PriorityDefault: return l.overloaded || len(l.waiters) >= l.cfg.MaxQueue default: // PriorityLow: no queueing at all once slots are exhausted return true } } // observeDelay tracks the minimum queueing delay per interval (CoDel-style). func (l *Limiter) observeDelay(d time.Duration) { now := time.Now() if now.After(l.intervalEnd) { l.overloaded = l.minDelay > l.cfg.TargetDelay l.minDelay = d l.intervalEnd = now.Add(l.cfg.Interval) return } if d < l.minDelay { l.minDelay = d } } func (l *Limiter) release(rtt time.Duration) { l.mu.Lock() defer l.mu.Unlock() l.inflight-- l.updateLimit(rtt.Seconds()) for len(l.waiters) > 0 && l.inflight < int(l.limit) { w := l.waiters[0] l.waiters = l.waiters[1:] w.granted = true l.inflight++ close(w.ready) } } // updateLimit is a simplified Gradient2-style update. func (l *Limiter) updateLimit(rtt float64) { if rtt <= 0 { return } if l.longRTT == 0 { l.longRTT, l.shortRTT = rtt, rtt } l.shortRTT += 0.2 * (rtt - l.shortRTT) l.longRTT += 0.01 * (rtt - l.longRTT) gradient := math.Max(0.5, math.Min(1.0, l.longRTT/l.shortRTT)) next := gradient*l.limit + math.Sqrt(l.limit) next = l.limit*(1-l.cfg.Smoothing) + next*l.cfg.Smoothing l.limit = math.Max(l.cfg.MinLimit, math.Min(l.cfg.MaxLimit, next)) } func (l *Limiter) removeWaiter(w *waiter) { for i, x := range l.waiters { if x == w { l.waiters = append(l.waiters[:i], l.waiters[i+1:]...) return } } } // Guard wraps a webhook handler with admission control. func Guard(l *Limiter, classify func(*http.Request) Priority, next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { release, ok := l.Acquire(classify(r)) if !ok { w.Header().Set("Retry-After", "10") http.Error(w, "service temporarily overloaded", http.StatusServiceUnavailable) return } start := time.Now() defer func() { release(time.Since(start)) }() next.ServeHTTP(w, r) }) }

Some limits of this sketch:

Waiters are served in FIFO order. Facebook’s design switches to LIFO once a queue forms, so the newest requests, which are most likely to still meet their sender’s deadline, go first.
RTT is measured for every request. In production you would typically exclude requests that failed fast.
A single mutex is fine at moderate rates. Profile before optimizing it.

6. Rejecting requests without losing events

Whether a rejected webhook is lost depends on the sender, so you need to know how each one behaves:

Senders with retries: Stripe retries with exponential backoff for up to 3 days in live mode. Shopify retries 8 times over 4 hours.
Senders without automatic retries: GitHub does not redeliver automatically. If you shed a GitHub delivery, you need to redeliver it manually or through the API once you recover.
Subscription risk: Shopify deletes a webhook subscription created through the Admin API after 8 consecutive failures, and Stripe emails you and eventually disables an endpoint that fails for multiple days. A long shedding episode can cost you the subscription itself.

For these reasons, shedding should be short, targeted and backed by a reconciliation job. Shopify itself recommends building one if you can’t afford to miss webhooks. The job periodically fetches data you may have missed through the provider’s API.

Status codes

503 Service Unavailable with Retry-After: the standard way to signal temporary overload. RFC 9110 defines Retry-After as either a number of seconds or an HTTP date.
429 Too Many Requests: RFC 6585 defines it for rate violations by a particular sender. It also accepts Retry-After.
Don’t just drop connections. An abrupt reset gives the sender no information, and it will treat the delivery as a failed attempt. A clean 503 tells it explicitly that the failure is temporary.

Do senders actually honor Retry-After? Some do. Webhook infrastructure vendors such as Hookdeck and Kit document that they respect Retry-After on 429 and 503, and AWS EventBridge API destinations are documented to as well. Svix advises that good senders will respect it when you return a 429. Stripe’s public documentation describes exponential backoff but doesn’t mention Retry-After, so check its behavior in your delivery logs rather than assuming. Because support varies, treat the header as a hint and rely on the sender’s own backoff as the fallback.

Add jitter to the values you return, for example 5 to 15 seconds instead of a constant 10. Otherwise senders that honor the header will retry in lockstep.

7. Priority-based shedding

Not all webhooks are equally urgent. A payment confirmation matters more than a bulk analytics sync. Classify by endpoint or authenticated partner, then apply different behavior per tier. The sketch above implements this policy:

Tier	Examples	Behavior in the sketch
`PriorityHigh`	Payment confirmations, security alerts	Queue for a slot; shed only when the wait queue is full.
`PriorityDefault`	Standard operational updates	Queue, but shed once the CoDel-style overload flag is set.
`PriorityLow`	Bulk syncs, analytics, non-essential notifications	Shed immediately when no slot is free.

This is a policy choice rather than a standard. Choose tiers based on which events you can safely replay later.

8. Observability

Since each node decides for itself, you need per-node metrics. Export them to Prometheus or OpenTelemetry:

webhook_ingress_concurrency_limit: gauge for the current adaptive limit.
webhook_ingress_inflight_requests: gauge for requests being processed.
webhook_ingress_shed_total{priority}: counter of rejected requests.
webhook_ingress_queue_delay_seconds: histogram of time spent waiting for a slot. Use millisecond-scale buckets, since the targets are in the single-digit-millisecond range.
webhook_ingress_overloaded: gauge (0 or 1) for the overload flag.

Alert on a sustained rise in shed counts or on queue delay staying above target. Both mean you should look at downstream capacity, not just the limiter.

9. Checklist

Acknowledge fast: Verify the signature, enqueue durably and return 2xx. Do the real work asynchronously.
Keep admission state local: Avoid a shared-store call on every inbound request.
Use a latency-driven limit: An adaptive, gradient-style limit beats a fixed number.
Watch queue delay, not just errors: Use a CoDel-style minimum-delay signal, and tune the target to your handler.
Reject explicitly: Return 503 (or 429 for per-sender limits) with a jittered Retry-After.
Know your senders: Document each provider’s timeout, retry window and subscription-deletion rules.
Reconcile: Run a job that backfills events missed during shedding, and redeliver GitHub deliveries manually.
Classify safely: Derive priority from routes or verified identity, never from unauthenticated headers.