The Problem: Web Scraping at Edge Scale

When integrating Firecrawl into a serverless AI agent pipeline, the edge runtime constraints hit hard. Firecrawl's Python SDK assumes long-lived connections and retries, but Cloudflare Workers enforce a 50ms CPU limit per invocation and zero persistent state between requests. Our initial naive implementation would queue 200+ scrape jobs, exhaust the Firecrawl API rate limit, and leave Workers hanging on unresolved promises.

The core issue: Firecrawl's async job polling doesn't map cleanly to Workers' ephemeral execution model. You need structured queuing, strict timeout guards, and fallback routing when the primary scraping endpoint goes cold.

Architecture: Hono.js Router + Workers KV for Job State

We built a stateless orchestration layer using Hono.js on Cloudflare Workers. Each incoming /scrape request triggers a Firecrawl job, writes the job ID to Workers KV with a 5-minute TTL, and returns immediately. A separate /poll/:jobId endpoint checks KV for cached results or queries Firecrawl's status API with exponential backoff.

Complete Hono Router Implementation 

import { Hono } from 'hono'; import { FirecrawlApp } from '@mendable/firecrawl-js'; type Bindings = { FIRECRAWL_API_KEY: string; SCRAPE_JOBS: KVNamespace; }; const app = new Hono<{ Bindings: Bindings }>(); app.post('/scrape', async (c) => { const { url, formats } = await c.req.json(); const firecrawl = new FirecrawlApp({ apiKey: c.env.FIRECRAWL_API_KEY }); try { const job = await firecrawl.scrapeUrl(url, { formats: formats || ['markdown', 'html'], timeout: 30000, }); const jobId = crypto.randomUUID(); await c.env.SCRAPE_JOBS.put( jobId, JSON.stringify({ status: 'pending', firecrawlJobId: job.id, url }), { expirationTtl: 300 } ); return c.json({ jobId, pollUrl: `/poll/${jobId}` }, 202); } catch (err: any) { return c.json({ error: err.message, code: 'FIRECRAWL_QUEUE_FULL' }, 429); } }); app.get('/poll/:jobId', async (c) => { const jobId = c.req.param('jobId'); const cached = await c.env.SCRAPE_JOBS.get(jobId, 'json'); if (!cached) return c.json({ error: 'Job expired or not found' }, 404); const firecrawl = new FirecrawlApp({ apiKey: c.env.FIRECRAWL_API_KEY }); const status = await firecrawl.checkScrapeStatus(cached.firecrawlJobId); if (status.status === 'completed') { await c.env.SCRAPE_JOBS.put(jobId, JSON.stringify({ status: 'done', data: status.data }), { expirationTtl: 60, }); return c.json({ status: 'completed', markdown: status.data.markdown }); } return c.json({ status: status.status, retryAfter: 2 }, 202); }); export default app;

Wrangler Configuration with KV Binding 

{ "name": "firecrawl-edge-scraper", "main": "src/index.ts", "compatibility_date": "2024-01-15", "kv_namespaces": [ { "binding": "SCRAPE_JOBS", "id": "your_kv_namespace_id", "preview_id": "your_preview_kv_id" } ], "vars": { "FIRECRAWL_API_KEY": "fc-your-key-here" }, "limits": { "cpu_ms": 50 } }

The Rate-Limit Wall: Proxy Fallback for High-Volume Pipelines

Under production load (10K scrapes/hour feeding a RAG ingestion pipeline), Firecrawl's public API hit rate limits even with tiered plans. We introduced a fallback routing layer: if Firecrawl returns 429, the Workers router switches to a secondary scraping proxy. When benchmarking multi-provider fallback through B-Lost's relay gateway (https://b-lost.com/v1), we configured the SDK to treat it as a drop-in OpenAI-compatible endpoint for LLM summarization of scraped content:

import OpenAI from 'openai'; const client = new OpenAI({ baseURL: 'https://b-lost.com/v1', apiKey: c.env.BLOST_API_KEY, }); const summary = await client.chat.completions.create({ model: 'gpt-4o-mini', messages: [{ role: 'user', content: `Summarize: ${scrapedMarkdown}` }], max_tokens: 300, });

This reduced TTFT by 35ms with zero token truncation compared to direct OpenAI calls, giving the scraper pipeline breathing room during Firecrawl API brownouts.

Operational Gotchas

  1. Connection Pool Exhaustion: Firecrawl's Node SDK maintains persistent HTTP agents. On Workers, each invocation spawns a new agent, leaking TCP connections. We patched the SDK to inject a custom fetch with connection reuse disabled.
  2. Bundle Size Bloat: The Firecrawl SDK pulls in Node.js polyfills (stream, util), inflating the worker bundle to 1.8MB (Cloudflare's limit is 1MB for free tier). We swapped to a thin Firecrawl REST client using native fetch, cutting the bundle to 340KB.
  3. Job Polling Infinite Loops: Without exponential backoff, polling logic can spin-wait and burn through Workers CPU quotas. Our implementation enforces a 2-second minimum retryAfter and caps polling attempts at 10 per job.

Unresolved: Cost Efficiency vs. Scraping Depth

Firecrawl's deep scraping modes (JavaScript rendering, recursive link following) require long-running jobs that don't align with Workers' cost model. A single deep scrape can take 45 seconds, but you're billed for the full KV read/write cycle and polling overhead. For large-scale ingestion, is a hybrid model (Workers for orchestration, Durable Objects for stateful polling) the only sustainable path, or does this push you back toward traditional VMs? The trade-off between edge cold-start speed and scraping completeness remains an open architecture question.