45.8s, then 12.6s: same prompt on Cloudflare Workers AI

I've been building a small tool that turns one technical post into an X thread, a LinkedIn post, a TikTok script and a newsletter blurb. It runs on Cloudflare Workers with Workers AI, on the free plan.

The product part was the easy half. Getting generation to behave predictably is what took me three passes — and the first lesson was that I couldn't even trust my own stopwatch.

Two requests, ten minutes apart, 3.6× difference

Both of these are real generations of the same sample post into the same two formats (a TikTok script plus a newsletter blurb), run within ten minutes of each other:

The only difference between them is a single character in the source. Same worker, same model, same batch, same region.

I can guess at the cause — a cold isolate, a busy shared queue, or both — but I can't tell them apart from the outside. What I can state for certain is that the spread is real, and that one sample tells you nothing.

(A day earlier the same batch measured 12.5 s. Same lesson, third number.)

Before that, my numbers were worse than useless: my local curl was resolving to an IPv6 address and hanging intermittently, producing 55–90 s results that had nothing to do with the server. Pinning curl -4 deleted all of them. Half of my first week of "performance problems" was my own tooling.

Why that spread is dangerous, not just annoying

Cloudflare closes a request at 100 seconds — the browser gets a 524 and any work still in flight is discarded. That's wall clock, not CPU time.

At 12 s per batch you have room for eight of them. At 45 s you have room for two. The same code that was comfortable on Monday is one slow queue away from a dead request on Tuesday.

This is the failure mode that made me redesign the request shape: not "how do I make it faster", but "how do I stop one slow batch from taking the whole run down with it".

Fix 1: batch by expected duration, not by user intent

My first version treated one run as one request — every selected format, await Promise.all(). The obvious problem is that a seven-format run then lives or dies as a single 100-second bet.

The browser now plans batches before sending anything. Cheap formats travel two at a time; each expensive one (X thread, LinkedIn post, long-form article) travels alone, and they're ordered so the visitor sees something usable while the slow ones are still working:

const SLOW_PLATFORMS = { x: 1, linkedin: 2, article: 3 }; const BATCH_MAX = 2; function planBatches(platforms) { const groups = []; const fast = platforms.filter((p) => !SLOW_PLATFORMS[p]); for (let i = 0; i < fast.length; i += BATCH_MAX) { groups.push(fast.slice(i, i + BATCH_MAX)); } // Sorted by weight so X lands before LinkedIn and the article comes last: // the visitor should see something usable while the slow ones are working. platforms .filter((p) => SLOW_PLATFORMS[p]) .sort((a, b) => SLOW_PLATFORMS[a] - SLOW_PLATFORMS[b]) .forEach((p) => groups.push([p])); return groups; } 

The cost is honesty about time: seven formats become five sequential requests, so the UI shows "batch 3 of 5" instead of one spinner. I'd rather show a number than show nothing.

Fix 2: one retry, with a reason

A single timeout shouldn't be a failed run. The same payload gets exactly one more attempt, and the user is told why they're waiting:

const postBatch = async (headers, batch, label) => { const attempt = () => { const ctl = new AbortController(); const timer = setTimeout(() => ctl.abort(), 90000); return fetch('/repurpose', { method: 'POST', headers, signal: ctl.signal, body: JSON.stringify({ text, platforms: batch, tone }) }).then((r) => r.json()).finally(() => clearTimeout(timer)); }; try { return await attempt(); } catch (e) { const transient = e && (e.name === 'AbortError' || e.name === 'TypeError'); if (!transient) throw e; hint.textContent = `Queue was busy on ${label} — retrying once…`; return await attempt(); } }; 

The retry only fires on transient failures. A 4xx quota refusal is not retried — repeating it just burns quota twice and reports the same thing.

What actually made it fast

Both fixes above are damage control. The real win came from an uncomfortable observation:

the most expensive prompt in my product was also the most repeated one.

The "try an example" button sends a fixed sample post with fixed formats. Those requests are byte-identical every single time — and every one of them was paying for a fresh 70B inference.

So generation now goes through Cloudflare's AI Gateway, which caches at the edge. It's free on every plan:

const r = await env.AI.run(MODEL, inputs, { gateway: { id: 'default', skipCache: false, cacheTtl: 3600 }, }); 

Same payload, measured before and after:

All three returned byte-identical output — which is itself the proof, since temperature: 0.7 cannot produce the same text twice by accident.

Two things I'd have wanted to know first:

• id: 'default' creates the gateway on your first authenticated request. No dashboard step, no extra API token.
• The cache key is a hash of the full request body, so it only helps when the input is truly identical. I checked the negative case too: changing one number in the source produced a fresh generation (12.6 s) with the corrected number in the output.

Three things I'd tell myself earlier

1. Measure twice, and pin your transport. My worst numbers were my own tooling, not the server. curl -4 turned a phantom 90-second stall into a real 6-second call.
2. On a shared queue, latency belongs to the platform, not to your code. Design for the spread, not for the average.
3. Cache the demo, not the product. Real inputs are unique. The button people click repeatedly is where your quota actually goes.

The tool is free and needs no card, if you'd rather see the output than read about it: https://feijihua.top

And the honest limit: it produces a skeleton, not your voice. I rewrote the opening line of the LinkedIn draft it gave me. That's the deal with a 70B model on a free queue.