Get the AI call out of your request handler

An API receives a PDF, asks a model to extract the fields, and returns the result. On your laptop it works: the model takes a few seconds, the response arrives, everyone is happy. The trouble starts when the second, the tenth and the hundredth user upload files at the same time.

People often describe this as "the API freezes." In Node.js that's only half true, and the other half changes the fix.

The event loop is (usually) not the villain

When your code awaits an HTTP call to an AI provider, Node.js does not sit and wait. The official Node.js guide explains that the event loop orchestrates client requests, and that the server stays fast as long as the work for each client is small. A network call is I/O: the process keeps serving other requests while the response is in flight.

The event loop really does block when the work is CPU-bound inside your own process: local OCR, image resizing, PDF generation, running a small model in JavaScript. For that, move the work out of the API process, period.

So if the API doesn't freeze, why change anything? Because the real problems are elsewhere:

1. Someone is holding the connection. The user, the proxy, the gateway and the load balancer all wait together, each with its own timeout.
2. The provider has limits. AI services cap concurrent requests, tokens per minute, or both. When you hit the cap, extra calls wait or fail, and the caller can't tell which.
3. Paid work gets thrown away. If the client gives up on a timeout, your server keeps calling the model. You pay for answers nobody reads.
4. Deploys and restarts lose work. A 20-second call in progress dies with the pod.
5. Nobody controls throughput or cost. Incoming traffic decides how many model calls run at once, not you.

What I measured in a small lab

To get past theory, I built two Node.js APIs calling a simulated provider: 3 seconds per call, 2 calls at a time. I fired 10 simultaneous requests, with the client giving up after 10 seconds, like a proxy would.

It's a simulation, not a benchmark, but the pattern is clear. And there's an important detail: the queue made nothing faster. The last result arrived around 15 seconds in both cases, because the bottleneck is the provider. What changed is who waits, who loses work, and who controls the pace.

Environment: Node.js 22, Redis 7, BullMQ 5, Express 4, run on November 14, 2026.

The design

Client ──POST /extract──▶ API ──add(job)──▶ Queue (Redis) ▲ │ │ │◀── 202 + /jobs/{id} ──┘ ▼ │ Worker(s) ──▶ AI provider └──GET /jobs/{id} (status/result) ◀── job state 

Five decisions make it work.

1. The API only accepts the request: 202 Accepted

RFC 9110 defines 202 as intentionally noncommittal: the request was accepted, but processing may not have started and may still fail. That's exactly the contract we want. The response tells the client where to check:

app.post('/extract', async (req, res) => { const key = req.get('Idempotency-Key') ?? randomUUID(); const job = await extractQueue.add('extract', { document: req.body.document }, { jobId: key }); const status = await job.getState(); // on a retried POST, returns the existing job's real state res.status(202).location(`/jobs/${job.id}`) .json({ jobId: job.id, status, statusUrl: `/jobs/${job.id}` }); }); 

2. A status endpoint (or a webhook)

The client polls GET /jobs/{id} until the state is completed or failed. While it's pending, the API sends Retry-After so the client doesn't hammer the endpoint. If the consumer is another system, a webhook on completion avoids polling.

app.get('/jobs/:id', async (req, res) => { const job = await extractQueue.getJob(req.params.id); if (!job) return res.status(404).json({ error: 'job not found' }); const state = await job.getState(); const body = { jobId: job.id, status: state, attemptsMade: job.attemptsMade }; if (state === 'completed') body.result = job.returnvalue; if (state === 'failed') body.error = job.failedReason; if (state !== 'completed' && state !== 'failed') res.set('Retry-After', '1'); res.json(body); }); 

3. Idempotency: the same request never becomes two jobs

Network timeouts make clients retry the POST. Without care, each retry triggers another paid model call. The Idempotency-Key header, being standardized at the IETF, solves this: the client sends one unique key per intent, and the server recognizes the repeats.

In the lab, the key becomes the jobId. BullMQ ignores an add with an ID that already exists, so three POSTs with the same key produced a single job. Your protection window is how long the job is kept (removeOnComplete), so set that window on purpose.

4. Retry with backoff, only for transient errors

export const extractQueue = new Queue('extract', { connection, defaultJobOptions: { attempts: 3, backoff: { type: 'exponential', delay: 1000 }, // 1s, 2s, 4s removeOnComplete: { age: 3600 }, // keep results (and dedup) for 1 hour removeOnFail: { age: 24 * 3600 }, }, }); 

A 503 or 429 from the provider deserves another try with growing delays. An invalid document doesn't: fail on the first attempt and record why. Retrying everything only multiplies the bill. In the lab, with 1 of every 3 provider calls failing, all six jobs completed; two needed three attempts.

5. A separate worker, with concurrency equal to your limit

const worker = new Worker('extract', async (job) => extract(job.data.document), { connection, concurrency: 2 }); for (const sig of ['SIGINT', 'SIGTERM']) { process.on(sig, async () => { await worker.close(); process.exit(0); }); } 

Worker concurrency is your cost and rate-limit control. Add up all replicas: the total should not exceed the limit you have with the provider. With API and worker separated, each one scales for its own reason: the API with request volume, the worker with queue depth. On Kubernetes that means two Deployments and, if it makes sense, an autoscaler watching queue length instead of CPU.

Shut the worker down gracefully, too: on SIGTERM it finishes the job in progress before exiting.

When NOT to do this

• The call takes under a second or two and volume is low. A queue brings Redis, a worker, monitoring and one more thing that can fail. Sometimes a well-tuned timeout and a concurrency limit inside the API are enough.
• The user needs a streamed response (chat). That's a different pattern: streaming with concurrency control and cancellation when the client leaves.
• You don't know yet whether the feature will stick. Start simple, measure, and change when the numbers ask for it.

Production checklist

• [ ] Endpoint returns 202 with Location or a status URL.
• [ ] Client sends Idempotency-Key; server deduplicates within a defined window.
• [ ] Retries only for transient errors, with backoff and a cap.
• [ ] Total worker concurrency ≤ provider limit.
• [ ] Failed jobs are monitored (how many, and why).
• [ ] Metrics: queue depth, wait time, processing time, cost per job.
• [ ] Graceful worker shutdown on deploy.
• [ ] Results have a retention period, and no sensitive data kept longer than needed.

The simulated provider (to reproduce the lab)

// Simulated AI/OCR provider: slow, limited concurrency, occasional failures. const LATENCY_MS = Number(process.env.PROVIDER_LATENCY_MS ?? 3000); const MAX_CONCURRENCY = Number(process.env.PROVIDER_MAX_CONCURRENCY ?? 2); const FAIL_EVERY = Number(process.env.PROVIDER_FAIL_EVERY ?? 0); let active = 0, calls = 0; const waiting = []; const acquire = () => active < MAX_CONCURRENCY ? (active++, Promise.resolve()) : new Promise((resolve) => waiting.push(resolve)); const release = () => { const next = waiting.shift(); next ? next() : active--; }; export async function extract(document) { await acquire(); try { calls++; await new Promise((r) => setTimeout(r, LATENCY_MS)); // I/O: does not block the event loop if (FAIL_EVERY > 0 && calls % FAIL_EVERY === 0) throw new Error('provider 503'); return { document, fields: { total: 42 } }; } finally { release(); } } 

The naive version is just res.json(await extract(req.body.document)) inside the route. Fire 10 concurrent requests at each with a 10-second client timeout and compare.

How do you treat AI calls in your APIs: as a regular request, or as background work?

References

• RFC 9110, §15.3.3 — 202 Accepted: https://www.rfc-editor.org/rfc/rfc9110.html#section-15.3.3
• Node.js — Don't Block the Event Loop (or the Worker Pool): https://nodejs.org/en/guides/dont-block-the-event-loop
• IETF draft — The Idempotency-Key HTTP Header Field: https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/
• BullMQ docs: https://docs.bullmq.io