Weekend Build Log: One In-Flight Request at a Time on a Free Model Endpoint

A weekend demo that calls a free model endpoint survives a refresh storm for one reason: identical prompts collapse into a single in-flight upstream call. Everything else — the UI, the storage, the prompt wording — is negotiable during a two-day build. Concurrency is not.

This is the build log for that layer. It covers what shipped, the two files that make the behaviour testable without a network, and the list of things that were deliberately skipped.

The failure mode worth designing out

Free-tier model endpoints are usually limited per account rather than per user, and a side project has exactly one effective user: whoever is clicking. Ten impatient refreshes are ten requests. A page that re-renders on every keystroke can quietly be hundreds.

Two small mechanisms bound that damage, and they solve different problems:

• Single-flight merges identical work that is already running. It saves tokens and reduces latency for everyone waiting.
• An admission cap limits how many upstream calls run at once. It keeps a burst from turning into a wall of rejects.

Neither is a queue, a cache, or a rate limiter. Mixing those up is how weekend projects grow a Redis dependency they never needed.

The scope contract

The build had a hard rule: anything not required to answer "does coalescing actually fire here?" was out of scope. That table is the whole project plan.

Artifact 1: single-flight, standard library only

No dependencies, no framework, about twenty lines that matter.

# singleflight.py """Collapse identical in-flight coroutines into one upstream call.""" import asyncio from typing import Any, Awaitable, Callable class SingleFlight: def __init__(self) -> None: self._inflight: dict[str, asyncio.Task] = {} self.coalesced = 0 self.upstream_calls = 0 async def do(self, key: str, factory: Callable[[], Awaitable[Any]]) -> Any: task = self._inflight.get(key) if task is None: self.upstream_calls += 1 task = asyncio.create_task(factory()) self._inflight[key] = task task.add_done_callback(lambda _t, k=key: self._inflight.pop(k, None)) else: self.coalesced += 1 # shield: one cancelled caller must not cancel work others are waiting on  return await asyncio.shield(task) 

Three details decide whether this works in practice:

1. The key must be stable. Normalize whitespace, strip volatile fields like timestamps or nonces, and include the model id and sampling parameters. A key containing time.time() never coalesces and fails silently — the code looks right and the counter stays at zero.
2. Retries belong inside the shared task. If the caller retries after a failure, every retry is a fresh upstream call. If the shared coroutine retries internally, one retry serves every waiter.
3. asyncio.shield changes the cancellation policy. All callers can walk away and the upstream call keeps running. That is intentional here — the result is still worth keeping — but it is a spend decision, not a free lunch. Cancel the task in the done callback if burned tokens matter more than a warm result.

Artifact 2: a burst test with no network

The test replaces the endpoint with asyncio.sleep. That makes it reproducible on a laptop, in CI, and on a plane, which is the point.

# demo.py import asyncio, json, time from singleflight import SingleFlight CAP = asyncio.Semaphore(2) # admission cap for the endpoint CALLS = 0 LATENCY = 1.2 # stand-in for endpoint latency, seconds  async def upstream(prompt: str) -> str: global CALLS async with CAP: CALLS += 1 await asyncio.sleep(LATENCY) return f"summary::{prompt}" async def main() -> None: sf = SingleFlight() prompts = [f"post-{i % 5}" for i in range(25)] # 25 requests, 5 distinct  t0 = time.perf_counter() out = await asyncio.gather(*(sf.do(p, lambda p=p: upstream(p)) for p in prompts)) wall = time.perf_counter() - t0 print(json.dumps({ "requests": len(prompts), "distinct_prompts": len(set(prompts)), "distinct_outputs": len(set(out)), "wall_seconds": round(wall, 2), "upstream_calls": CALLS, "coalesced": sf.coalesced, }, indent=2)) if __name__ == "__main__": asyncio.run(main()) 

Run it with python demo.py. Nothing leaves the machine.

Reading the output

The expected shape is arithmetic, not measurement: 25 requests over 5 distinct prompts, capped at 2 concurrent upstream calls, gives 3 waves of 1.2s — roughly 3.6s of wall time, 5 upstream calls, and 20 coalesced requests. The script prints its own numbers; treat any figure quoted in a blog post, including this one, as a design target until the counters agree.

The useful signal is upstream_calls == distinct_prompts. If it is not, the key is wrong. That single assertion catches more single-flight bugs than any benchmark.

Where free model access and a free server changed the design

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The build targets MonkeyCode's free model access and free server option, which the operator describes as a substantial monthly token allowance plus a hosted server at no cost; the exact quota, eligibility, and any expiry rules are stated on the product page and should be read before planning around them, because those numbers move.

Free availability shaped three decisions that paid hosting would not have forced:

• A cap of 2, not 32. A shared free endpoint is not a place to test how well your code handles concurrency. Serialize first; raise the cap only when the demo needs it.
• State in memory. A free server may restart without warning, so anything durable was cut. The single-flight map is allowed to evaporate.
• No long-running worker. Work completes inside the request. A background queue on an instance that can be reclaimed is a bug with a delay fuse.

If the demo outgrows those constraints, the right move is to revisit hosting, not to bolt a database onto a free instance.

The skip list

The honest part of a build log is the list of things that are missing on purpose.

1. Streaming responses. Coalescing a token stream means picky fan-out semantics. Skipped until a second user appears.
2. A TTL cache on top of single-flight. Different problem: single-flight dedupes concurrent work, a cache dedupes sequential work. Both are useful; only one was needed for the demo.
3. Per-key fairness. With a cap of 2, a slow prompt can hold a slot. Fine at demo scale, not fine at ten users.
4. Observability beyond one JSON line. No OpenTelemetry, no dashboards. The counters print to stdout and that was enough to catch two key-normalization mistakes.
5. Auth. Single user, private URL, weekend scope.

Limitations and who should skip this

Single-flight does nothing when prompts are unique. A demo that summarizes a different document on every request gets zero benefit and one extra dictionary lookup.

The map is per process. Two instances behind a load balancer coalesce nothing, and the code will look like it is working.

A cap of 1 introduces head-of-line blocking; a cap of 2 halves it and doubles burst tolerance. Neither is free, and neither is a rate limiter.

Skip this approach if the project needs per-tenant quotas, durable job execution, or streamed output. Skip it if correctness of the model output is the open question — coalescing makes wrong answers arrive faster and cheaper.

What ships at the end of the weekend

A demo, five tests, and a rule that survives contact with real traffic: identical work in flight is wasted work. The interesting engineering on a free tier is mostly subtraction, and subtraction is testable.

If you want to run this against a real endpoint instead of asyncio.sleep, the free model access and free server option are the cheapest place to start — swap upstream() for the client call and keep the burst script unchanged.