The problem

Your load test says p99 latency is 12ms. You ship it. Three weeks later, real users are complaining about multi-second hangs, and your dashboards — fed by the same percentile math — say everything's fine. This isn't a monitoring gap. It's a measurement bias baked into almost every load-testing tool you've ever used, and it has a name: coordinated omission.

Here's the setup that causes it. Most load generators work in a closed loop: send a request, wait for the response, record the latency, send the next request. That's how ab, naive wrk scripts, and a lot of homegrown load test harnesses work by default. It feels correct — you're measuring exactly how long each request took.

Why it happens

The problem is what happens during a stall. Say your service has a GC pause, a lock contention spike, or a downstream timeout that freezes request processing for 1 second. In a closed-loop generator targeting 100 req/s, during that 1-second stall you issue exactly one request — the one that's already in flight. It comes back slow, and you record one bad sample: 1000ms.

But that's not what your real traffic does. Real clients don't politely wait their turn. If your service normally handles 100 req/s, a 1-second stall means roughly 100 requests should have arrived and are now queued up behind the one you're measuring. Your closed-loop tool never issued them — it was still blocked waiting on the first one. Those 99 "missing" requests, each of which would have experienced close to a full second of delay, simply never get sampled. They're omitted from your percentile calculation, and the omission is coordinated with the stall itself — it happens exactly when the result would've been ugliest.

The term comes from Gil Tene (Azul Systems, creator of HdrHistogram), originally describing how JVM pause measurement tools lied about GC impact. It generalizes to any closed-loop benchmark of any system: HTTP APIs, databases, queues, LLM inference endpoints, all of it.

The numbers get dramatic fast. Correcting a closed-loop measurement for omission — recomputing what the actual arrival-rate-based percentiles would have been — routinely turns a reported p99 in the single-digit milliseconds into a real p99 north of several hundred milliseconds to a full second. The pain isn't a rounding error. It's an order-of-magnitude lie, concentrated exactly in the tail you care about.

What to do about it

Three concrete fixes, in order of effort:

  1. Use an open-loop (or corrected) load generator. Tools like wrk2, k6 (with arrival-rate executors), Gatling, and Locust's constant-arrival-rate shape issue requests on a fixed schedule regardless of whether the previous one has returned. That matches how production traffic actually behaves — users don't queue politely behind your slow request.

  2. Record "intended" vs. "actual" send time, and correct for it. If you can't switch tools, at minimum log when a request should have been sent versus when it was sent, and compute corrected percentiles. HdrHistogram has built-in support for this via its coordinated-omission correction methods.

  3. Treat any closed-loop percentile number as a lower bound, never a ceiling. If your load test says p99 is 12ms, the honest statement is "p99 is at least 12ms under ideal, non-stalled conditions" — not "p99 is 12ms." That reframing alone changes how much you should trust a green load test before a launch.

This matters more, not less, in the agentic/LLM era: inference endpoints have wildly variable per-request latency (cache hits vs. cold starts, short vs. long generations), which means stalls are frequent and your closed-loop load test is actively hiding your worst-case behavior right when you need to see it.

Key takeaways

  • Closed-loop load generators under-sample exactly the requests that would show your worst latency — the omission is coordinated with the stall, not random.
  • A reported p99 from a closed-loop tool is a floor, not a real number. The true tail can be an order of magnitude worse.
  • Switch to open-loop / constant-arrival-rate tools (wrk2, k6 arrival-rate, Gatling, Locust) or correct for omission with HdrHistogram.
  • This bites hardest on systems with high latency variance — which includes almost every LLM-backed service shipping today.