A 90-second upstream blip doesn't take you down. What your retries do in the ninety seconds after it ends does.

We learned this the expensive way. A vendor API returned 502s for about a minute and a half. Three thousand executions were queued behind it across 40 workers. When the vendor recovered, every worker fired at the same cadence, one second apart, forever synchronized, because our retry loop was sleep(1) with no jitter. p99 went 200ms → 18s. The vendor's rate limiter banned our egress IP for ten minutes. Our Postgres pool of 20 connections was gone, held by executions parked on socket timeouts.

Three separate defects, none of them in the vendor's code:

  1. Correlated retries. Fixed-interval backoff keeps a herd in lockstep. It never decorrelates.
  2. No shared failure state. Each worker independently concluded the upstream was healthy.
  3. No cap on aggregate in-flight work. Retries amplified load exactly when the downstream was weakest.

What n8n gives you, and where it stops

n8n's node-level Retry On Fail (Max Tries / Wait Between Tries) is a fixed delay, applied per item, inside one worker process. In queue mode (EXECUTIONS_MODE=queue) that means retry state is per-process and per-execution: no jitter, no cross-worker breaker, and no way to distinguish "this item failed" from "this dependency is down." Continue On Fail keeps the workflow moving; it does not stop the fan-out. The fix has to live in the HTTP client your Code node calls, not in node settings.

The coarse safety valve is worker concurrency: n8n worker --concurrency=10 × 40 workers = 400 in-flight ceiling. That number has to be one the downstream will tolerate, not one your CPU can handle.

Deliverable 1 — full-jitter backoff plus a breaker that actually opens 

import random, time, httpx RETRYABLE = {429, 500, 502, 503, 504} class Breaker: def __init__(self, threshold=5, reset_after=30.0): self.threshold, self.reset_after = threshold, reset_after self.fails, self.opened_at = 0, 0.0 def allow(self): if self.fails < self.threshold: return True if time.monotonic() - self.opened_at >= self.reset_after: self.fails = self.threshold - 1 # half-open: one probe through  return True return False def record(self, ok): if ok: self.fails = 0 return self.fails += 1 if self.fails >= self.threshold: self.opened_at = time.monotonic() def request(client, method, url, *, attempts=5, base=0.4, cap=20.0, breaker=None, **kw): err = None for i in range(attempts): if breaker and not breaker.allow(): raise RuntimeError(f"circuit open: {url}") try: r = client.request(method, url, **kw) except (httpx.ConnectError, httpx.ConnectTimeout, httpx.ReadTimeout) as exc: if breaker: breaker.record(False) err = exc else: if breaker: breaker.record(r.status_code < 500) if r.status_code not in RETRYABLE: return r err = r.status_code if r.status_code == 429 and (ra := r.headers.get("Retry-After")): time.sleep(min(float(ra), cap)) continue if i == attempts - 1: break time.sleep(random.uniform(0, min(cap, base * 2 ** i))) # full jitter  raise RuntimeError(f"exhausted {attempts} attempts: {err}")

Full jitter (random(0, min(cap, base·2^i))) flattens the herd into a smooth arrival curve instead of the multi-modal spike decorrelated jitter produces. Retry-After wins over your schedule when present. And never retry a POST without an idempotency key at the transport layer — that's how you get double charges, not resilience.

Deliverable 2 — exact sliding window in one atomic Lua eval

Check-and-increment must be atomic. INCR + EXPIRE is two round trips and leaks counters on crash; fixed-window counters permit a 2× burst across the boundary. A sorted set is exact:

-- KEYS[1] = window key ARGV = now_ms, window_ms, limit, member local now = tonumber(ARGV[1]) local window = tonumber(ARGV[2]) local limit = tonumber(ARGV[3]) redis.call('ZREMRANGEBYSCORE', KEYS[1], 0, now - window) if redis.call('ZCARD', KEYS[1]) >= limit then local oldest = redis.call('ZRANGE', KEYS[1], 0, 0, 'WITHSCORES') return {0, tonumber(oldest[2]) + window - now} end redis.call('ZADD', KEYS[1], now, ARGV[4]) redis.call('PEXPIRE', KEYS[1], window) return {1, 0} 
import time, uuid, redis.asyncio as redis from fastapi import FastAPI, Request from fastapi.responses import JSONResponse app = FastAPI() r = redis.Redis(host="127.0.0.1", port=6379, decode_responses=True) window = r.register_script(open("sliding_window.lua").read()) @app.middleware("http") async def rate_limit(request: Request, call_next): tenant = request.headers.get("x-tenant", "anon") key = f"{{rl:{tenant}}}:{request.url.path}" # hash tag = cluster-safe  allowed, retry_ms = await window( keys=[key], args=[int(time.time() * 1000), 60_000, 120, uuid.uuid4().hex], ) if not allowed: return JSONResponse( {"error": "rate_limited"}, status_code=429, headers={"Retry-After": str(max(1, int(retry_ms / 1000)))}, ) return await call_next(request)

One member per request inside the window is the storage cost, bounded by the limit itself. The {rl:tenant} hash tag keeps a tenant's keys in one Redis Cluster slot — drop it and EVAL fails cross-slot under Cluster.

Wiring it into a workflow

A Code node cannot hold breaker state across items, and in queue mode there are N worker processes anyway — in-memory state is a lie. Put the breaker in Redis and call the wrapper from a sidecar service your workflow hits, rather than reimplementing the logic per workflow. Then set Retry On Fail to Max Tries = 1 on the node itself. Node retry × client retry multiplies: 3 × 5 attempts is fifteen calls per item, which is the amplification that started this.

After the change: retries spread across 0–20s, peak arrival rate down roughly 8×, no ban, 60% pool headroom at steady state. The tradeoff you inherit is that an open breaker fails fast and silent — surface it as its own alert state, distinct from generic node errors, or you'll drop work while the dashboard stays green.