Inbound webhooks are deceptively hard to handle at scale. Providers such as Stripe, GitHub, and Shopify don’t know about your database connection pool, your autoscaling lag, or a brief microservice outage. They send events when things happen, and a busy period for one of your customers can turn into a burst of requests against your origin.

If the origin sits directly behind a load balancer, one of those bursts can exhaust database connections and memory and cause timeouts. Some senders then retry, which adds more load while you are already struggling.

One way to protect the origin is an ingress buffer at the edge. With Cloudflare Workers, Workers KV, and Durable Objects, you can verify signatures, deduplicate, rate-limit, and durably queue incoming webhooks before a single request reaches your infrastructure. Cloudflare runs Workers in more than 330 cities and says that puts code within roughly 50 ms of about 95% of the world’s Internet-connected population.

One caveat up front: only the entrypoint Worker is “everywhere”. Each tenant’s Durable Object is a single globally unique instance that coordinates all requests sent to it. The buffer is therefore distributed across tenants, not replicated for any one tenant.

Why Webhook Ingestion Fails

Each of these failure modes shapes the design below.

  1. Retry behavior differs by provider. Stripe retries failed live-mode deliveries with exponential backoff for up to three days and disables endpoints that keep failing. Shopify retries 8 times over the following 4 hours. GitHub does not automatically redeliver failed deliveries at all. A short outage can therefore produce a burst of retries from some senders and permanently lost events from others. You can’t assume one retry contract.
  2. Senders give up quickly. Shopify allows a one-second connection timeout and a five-second timeout for the whole request. If your origin does real work before responding, slow responses are treated as failures and trigger the retries you were trying to avoid.
  3. Duplicates are normal. Providers deliver at least once. Shopify says its app “might receive the same webhook more than once”, for example after a network timeout or a retry, and recommends deduplicating on the X-Shopify-Webhook-Id header. Stripe events carry a stable event ID that stays the same across retries.
  4. One tenant can starve the rest. A misconfigured integration that fires thousands of events can consume shared processing capacity unless you isolate and limit it per tenant.

Acknowledging quickly after a durable write handles the first two, deduplication handles the third, and a per-tenant limiter handles the fourth.

Architecture 

[Webhook sender] │ ▼ [Worker: validate tenant, verify HMAC] │ │ │ └──► [Workers KV: cached tenant config, secrets, limits] ▼ [Durable Object (one per tenant, SQLite-backed)] ├─ dedupe by event ID ├─ per-tenant rate limit ├─ durable event queue └─ alarm → batched delivery to origin (or to Cloudflare Queues)
  • Worker (entrypoint): Parses the path, loads tenant config, verifies the signature, and forwards verified events to the tenant’s Durable Object. Anycast routing and Cloudflare’s scheduling typically place it close to the sender.
  • Workers KV: A read-optimized, eventually consistent cache for data that changes rarely, such as tenant config and rate limits. Cloudflare cites sub-5 ms hot read latency. Writes can take 60 seconds or more to be visible in other locations.
  • Durable Objects: Provide per-tenant coordination with a private, transactional SQLite database. Because each object processes one tenant’s events, dedupe checks and rate-limit counters don’t need distributed locks.
  • Alarms: Each object can schedule a single alarm, which lets it drain its buffer in batches without waiting for another incoming request.

Step 1: The Entrypoint Worker

The Worker reads the raw bytes, verifies the HMAC over exactly those bytes, and only then touches the Durable Object. Several details differ from naive implementations:

  • Fail closed. If there is no secret for the tenant, the request is rejected. A shared fallback secret would let anyone who knows it forge events for every tenant.
  • Constant-time comparison. Workers provide crypto.subtle.timingSafeEqual. Cloudflare’s documentation notes it is not constant-time with respect to length, which is harmless here because an HMAC-SHA256 digest is always 32 bytes.
  • Meaningful dedupe keys. If the sender provides no event ID, the code falls back to a hash of the body. A random UUID would defeat deduplication, because every retry would look new.
  • Minimal forwarding. Only what the origin needs is passed on, not every header. Forwarding all headers would persist signatures and other sensitive values. 
// src/index.ts import type { TenantWebhookBuffer } from "./tenant-webhook-buffer"; export interface Env { TENANT_DO: DurableObjectNamespace<TenantWebhookBuffer>; WEBHOOK_KV: KVNamespace; ORIGIN_BASE_URL: string; // e.g. https://api.example.com ORIGIN_TOKEN: string; // set with `wrangler secret put ORIGIN_TOKEN` } interface TenantConfig { active: boolean; rateLimit: number; // max accepted events per 10-second window secrets: string[]; // current + previous secret during rotation } const TENANT_ID = /^[A-Za-z0-9_-]{1,64}$/; const encoder = new TextEncoder(); export default { async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> { if (request.method !== "POST") { return new Response("Method Not Allowed", { status: 405, headers: { Allow: "POST" } }); } // Expected path: /webhook/<tenantId> const [, prefix, tenantId] = new URL(request.url).pathname.split("/"); if (prefix !== "webhook" || !tenantId || !TENANT_ID.test(tenantId)) { return new Response("Not Found", { status: 404 }); } const config = await getTenantConfig(env, ctx, tenantId); if (!config?.active) { return new Response("Unknown tenant", { status: 404 }); } // Verify against the exact bytes the sender signed. const bodyBytes = new Uint8Array(await request.arrayBuffer()); const signature = request.headers.get("X-Signature") ?? ""; let valid = false; for (const secret of config.secrets) { if (await verifyHmacHex(bodyBytes, signature, secret)) valid = true; } if (!valid) { return new Response("Invalid signature", { status: 401 }); } const eventId = request.headers.get("X-Event-ID") ?? (await sha256Hex(bodyBytes)); try { const stub = env.TENANT_DO.getByName(tenantId); const result = await stub.ingestWebhook( { eventId, receivedAt: Date.now(), contentType: request.headers.get("Content-Type") ?? "application/octet-stream", rawBody: new TextDecoder().decode(bodyBytes), // assumes UTF-8 payloads (JSON) }, config.rateLimit, ); return Response.json(result.body, { status: result.status, headers: result.retryAfter ? { "Retry-After": String(result.retryAfter) } : undefined, }); } catch { // The event was NOT durably stored, so ask the sender to retry. return new Response("Temporarily unavailable", { status: 503, headers: { "Retry-After": "30" } }); } }, }; async function verifyHmacHex(body: Uint8Array, headerSig: string, secret: string): Promise<boolean> { const provided = hexToBytes(headerSig); if (!provided) return false; const key = await crypto.subtle.importKey( "raw", encoder.encode(secret), { name: "HMAC", hash: "SHA-256" }, false, ["sign"], ); const expected = new Uint8Array(await crypto.subtle.sign("HMAC", key, body)); return provided.length === expected.length && crypto.subtle.timingSafeEqual(provided, expected); } function hexToBytes(hex: string): Uint8Array | null { if (hex.length % 2 !== 0 || !/^[0-9a-fA-F]*$/.test(hex)) return null; const out = new Uint8Array(hex.length / 2); for (let i = 0; i < out.length; i++) out[i] = parseInt(hex.slice(i * 2, i * 2 + 2), 16); return out; } async function sha256Hex(data: Uint8Array): Promise<string> { const digest = await crypto.subtle.digest("SHA-256", data); return Array.from(new Uint8Array(digest), (b) => b.toString(16).padStart(2, "0")).join(""); }

This example assumes a generic sender that sends a hex HMAC-SHA256 of the body in X-Signature. Real providers differ, so adapt the verification step:

  • GitHub sends sha256=<hex> in X-Hub-Signature-256.
  • Shopify sends a base64-encoded digest in X-Shopify-Hmac-SHA256.
  • Stripe signs a timestamp plus the payload, which also gives you replay protection.

For providers with a stable delivery ID, use it as the dedupe key: X-Shopify-Webhook-Id for Shopify, X-GitHub-Delivery for GitHub, and the event id in the body for Stripe.

The Worker calls the Durable Object through Workers RPC. Public methods on a class that extends DurableObject are callable through a stub. Arguments and return values must be serializable, so this example returns a plain object and builds the Response in the Worker.

Step 2: Per-Tenant State in a Durable Object

Each tenant’s Durable Object holds four things in its SQLite database:

  • processed event IDs, for deduplication
  • a short rate-limit window
  • the pending event queue
  • a small counter for delivery backoff

SQLite-backed Durable Objects are Cloudflare’s recommended storage backend for new classes, and the SQL API is synchronous. Writes made without an intervening await are committed together, and an output gate holds the response until the writes are confirmed. The 202 therefore means the event has been durably stored, not just accepted into memory.

Keeping deduplication and rate limiting in separate tables matters. Pruning old rate-limit rows from the same table that records processed IDs would erase your dedupe history after seconds.

// src/tenant-webhook-buffer.ts import { DurableObject } from "cloudflare:workers"; import type { Env } from "./index"; export interface WebhookEvent { eventId: string; receivedAt: number; contentType: string; rawBody: string; } export interface IngestResult { status: 200 | 202 | 429; body: Record<string, unknown>; retryAfter?: number; } const RATE_WINDOW_MS = 10_000; const DEDUPE_RETENTION_MS = 4 * 24 * 60 * 60 * 1000; // longer than Stripe's 3-day retry window const BATCH_SIZE = 50; const FLUSH_DELAY_MS = 2_000; const MAX_BACKOFF_MS = 5 * 60 * 1000; export class TenantWebhookBuffer extends DurableObject<Env> { constructor(ctx: DurableObjectState, env: Env) { super(ctx, env); ctx.blockConcurrencyWhile(async () => { const sql = ctx.storage.sql; sql.exec(`CREATE TABLE IF NOT EXISTS processed_events ( event_id TEXT PRIMARY KEY, received_at INTEGER NOT NULL)`); sql.exec(`CREATE TABLE IF NOT EXISTS rate_window (received_at INTEGER NOT NULL)`); sql.exec(`CREATE INDEX IF NOT EXISTS idx_rate_window ON rate_window (received_at)`); sql.exec(`CREATE TABLE IF NOT EXISTS event_queue ( id INTEGER PRIMARY KEY AUTOINCREMENT, event_id TEXT NOT NULL, payload TEXT NOT NULL, received_at INTEGER NOT NULL)`); sql.exec(`CREATE TABLE IF NOT EXISTS flush_state ( id INTEGER PRIMARY KEY CHECK (id = 1), failures INTEGER NOT NULL DEFAULT 0)`); sql.exec(`INSERT OR IGNORE INTO flush_state (id, failures) VALUES (1, 0)`); }); } async ingestWebhook(event: WebhookEvent, limit = 100): Promise<IngestResult> { const sql = this.ctx.storage.sql; const now = Date.now(); // 1. Deduplicate. Duplicates are acknowledged with 200 and cost almost nothing. const seen = sql.exec("SELECT 1 FROM processed_events WHERE event_id = ?", event.eventId).toArray(); if (seen.length > 0) { return { status: 200, body: { status: "deduplicated", eventId: event.eventId } }; } // 2. Sliding-window rate limit over accepted events. sql.exec("DELETE FROM rate_window WHERE received_at < ?", now - RATE_WINDOW_MS); const { count } = sql.exec<{ count: number }>("SELECT COUNT(*) AS count FROM rate_window").one(); if (count >= limit) { return { status: 429, body: { status: "rate_limited" }, retryAfter: 10 }; } // 3. Record and enqueue atomically. this.ctx.storage.transactionSync(() => { sql.exec("INSERT INTO processed_events (event_id, received_at) VALUES (?, ?)", event.eventId, now); sql.exec("INSERT INTO rate_window (received_at) VALUES (?)", now); sql.exec( "INSERT INTO event_queue (event_id, payload, received_at) VALUES (?, ?, ?)", event.eventId, JSON.stringify(event), now, ); }); // Schedule a flush only if one isn't pending. Calling setAlarm() again would // overwrite the pending alarm and keep pushing it back under steady traffic. if ((await this.ctx.storage.getAlarm()) === null) { await this.ctx.storage.setAlarm(now + FLUSH_DELAY_MS); } return { status: 202, body: { status: "buffered", eventId: event.eventId } }; } async alarm(): Promise<void> { const sql = this.ctx.storage.sql; // Housekeeping: expire old dedupe records. sql.exec("DELETE FROM processed_events WHERE received_at < ?", Date.now() - DEDUPE_RETENTION_MS); const rows = sql .exec<{ id: number; payload: string }>("SELECT id, payload FROM event_queue ORDER BY id LIMIT ?", BATCH_SIZE) .toArray(); if (rows.length === 0) return; let delivered = false; try { const response = await fetch(`${this.env.ORIGIN_BASE_URL}/internal/webhooks/batch`, { method: "POST", headers: { "Content-Type": "application/json", Authorization: `Bearer ${this.env.ORIGIN_TOKEN}`, }, body: JSON.stringify({ events: rows.map((r) => JSON.parse(r.payload)) }), signal: AbortSignal.timeout(10_000), }); delivered = response.ok; } catch { delivered = false; } if (delivered) { sql.exec("DELETE FROM event_queue WHERE id <= ?", rows[rows.length - 1].id); sql.exec("UPDATE flush_state SET failures = 0 WHERE id = 1"); const { n } = sql.exec<{ n: number }>("SELECT COUNT(*) AS n FROM event_queue").one(); if (n > 0) await this.ctx.storage.setAlarm(Date.now() + FLUSH_DELAY_MS); // keep draining } else { const { failures } = sql.exec<{ failures: number }>("SELECT failures FROM flush_state WHERE id = 1").one(); sql.exec("UPDATE flush_state SET failures = ? WHERE id = 1", failures + 1); const delay = Math.min(FLUSH_DELAY_MS * 2 ** failures, MAX_BACKOFF_MS); await this.ctx.storage.setAlarm(Date.now() + delay); } } }

A few design points are worth expla...