When Should an LLM Router Stop Being Sticky? llm-d 1.0 Cache-Affinity Routing, Break-Even Math, and a Simulation

TL;DR

• What shipped: llm-d 1.0 was announced on 2026-10-20 under CNCF governance, with vLLM and SGLang as supported model servers. The feature that most affects your GPU bill is not the headline list. It is the default routing rule described in llm-d's August post "Sticky Until Saturated": send a request to the endpoint that already holds its prefix, until that endpoint's queued token load passes a limit, then route by load alone.
• The limit is arithmetic, not taste: tau = R_peak x T_max. With the published Qwen3-32B numbers (8,192 tokens in 0.40 s, so 20,480 tokens/s, and a 14 s tolerance) that gives 286,720 tokens, exactly 35 scheduler chunks.
• A break-even you can compute: for one request, sticking beats spilling while the warm endpoint's queue exceeds the cold one's by no more than the tokens the cache saves. For a 55,000-token prefix that is 2.69 s of queue. llm-d's limit is 5.2 times larger on purpose, and my simulation shows what happens if you set it at break-even instead: the fleet collapses.
• Reported results (vendor benchmark): 2.9x and 2.0x the throughput of Kubernetes Service round-robin on two prefill-bound workloads, parity on a decode-bound one.

What did llm-d 1.0 actually ship?

The announcement lists a lot: prefix-cache and load-aware routing, KV cache offloading, prefill/decode disaggregation, wide expert parallelism, batch and async inference, multi-tenant quality of service, router high availability with hot standby, and session-aware routing for agents. The founding contributors named are Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA, with more than 500 contributors from over 25 organizations. The GitHub release notes add a few operational details: a DisaggregatedSet that manages prefill and decode replicas as one logical unit, a router configuration API that graduates to v1, and deprecation of the llm-d-rocm container image builds in favor of upstream vLLM.

Most of that is plumbing you will only appreciate during an incident. The routing default is different, because it decides how often your GPUs recompute a prompt prefix they have already seen. If you serve long shared system prompts (coding agents, RAG with a fixed preamble, multi-turn chat), prefix reuse is the largest single lever on cost per token, and the router is the component that either preserves it or throws it away.

IBM's announcement also quotes figures such as 11.5 times lower inference cost than an all-proprietary configuration at roughly a 98% prefix-cache hit rate. Those are the project backers' numbers with workload details I could not check, so I treat them as a statement of how much headroom a high hit rate can give, not as something to plan capacity around.

Why does cache-affinity routing hot-spot?

Round-robin ignores caches. A fleet of 10 endpoints, each able to hold only a few long prefixes, ends up with every endpoint seeing every prefix, evicting constantly, and recomputing most of what it receives. Affinity routing fixes that by sending a request to an endpoint that already has its prefix.

The failure mode is the other direction. In llm-d's own words, affinity concentrates traffic on cache-warm endpoints, and without an explicit saturation release the warm endpoint keeps absorbing load past the point where a cold endpoint would serve the request faster. Real traffic is skewed. A few popular prefixes carry a large share of requests, so the endpoint that owns the most popular prefix becomes a queue.

The previous llm-d default tried to handle this with a four-signal weighted blend: a prefix-cache scorer at weight 3, plus queue, KV-cache-utilization and no-hit-LRU scorers at weight 2 each. The authors describe the behavior as hard to predict and harder to tune. That is a fair description of any weighted sum where one signal is in tokens, one is a count and one is a ratio: the weights encode an exchange rate between units nobody measured.

How does llm-d decide when to stop being sticky?

The replacement splits the decision in two. A filter keeps only endpoints whose cached prefix match is at or above an affinityThreshold of 0.8. Then one load scorer, matched to the workload's bottleneck, picks among them: token-load-scorer (uncached prefill tokens in flight) for prefill-bound traffic, active-request-scorer (active request count) for decode-bound traffic. When the best warm endpoint's load passes the saturation limit tau, the filter stops restricting and the request goes wherever load is lowest.

The limit comes from two things you can measure:

tau = R_peak x T_max R_peak = peak prefill throughput per endpoint (tokens/s) T_max = how much TTFT degradation you will tolerate 

For the benchmark setup (Qwen3-32B, tensor parallel 2 across two H100s per server, vLLM with max-num-batched-tokens=8192), the post reports T(B = 8192) = 0.40 s. I checked the rest of the chain: 8,192 / 0.40 = 20,480 tokens/s; 20,480 x 14 s = 286,720; and 286,720 / 8,192 = 35.0 chunks. The numbers are internally consistent.

Here is the same calculation as code, with the break-even and the KV sizing I use below. The KV geometry comes from the model's config.json on Hugging Face (64 layers, 8 KV heads, head dimension 128, bf16):

def saturation_tau(batch_tokens, batch_seconds, ttft_tolerance_s): r_peak = batch_tokens / batch_seconds # prefill tokens/s per endpoint  return r_peak, int(r_peak * ttft_tolerance_s) def kv_bytes_per_token(layers, kv_heads, head_dim, dtype_bytes=2): return 2 * layers * kv_heads * head_dim * dtype_bytes # K and V  r, tau = saturation_tau(8192, 0.40, 14) print(r, tau, tau / 8192) # 20480.0 286720 35.0 per_tok = kv_bytes_per_token(64, 8, 128) print(per_tok, 55000 * per_tok / 2**30) # 262144 bytes, 13.43 GiB per 55k prefix print(int(64 * 2**30 // (55000 * per_tok))) # 4 prefixes fit in a 64 GiB budget 

That last line matters. At 256 KiB per token, one 55,000-token prefix is 13.4 GiB of KV. A server with a 64 GiB KV budget (my assumption, not a figure from the post) holds four of them. A coding-agent fleet with dozens of distinct long prefixes cannot cache them all on every endpoint, which is exactly why routing decides the hit rate.

What throughput did the benchmark report?

The post compares matched configurations against Kubernetes Service round-robin on 10 Qwen3-32B servers (8 for the third workload). All figures below are from llm-d's own write-up, not an independent test.

The pattern is the useful part. Where prefill dominates and prefixes are long, affinity roughly doubles to triples capacity. Where decode dominates, there is little prefill to save, so throughput is at parity and the only gain is a later knee. The same routing rule is not a universal win, and the post's choice to pair the load signal with the bottleneck is the honest way to say so.

What is the break-even point for stickiness?

Take one request whose prefix of P tokens is cached on endpoint A and absent from endpoint B. It carries U new tokens. On A it costs U tokens of prefill after waiting q_A tokens in queue. On B it costs P + U tokens after waiting q_B. Sticking is faster for that request while

q_A + U <= q_B + P + U => q_A - q_B <= P 

The queue gap you should accept is exactly the number of tokens the cache saves. For P = 55,000 at 20,480 tokens/s that is 2.69 s. A cached request costs 73 ms of prefill; a cold one costs 2.76 s, a ratio of 37.7.

So llm-d's 14 s tolerance is not the per-request break-even. It is 5.2 times larger. That looks wrong until you count what a spill costs beyond the one request. A cold fill burns 2.76 s of the receiving endpoint's capacity, and it inserts a 13.4 GiB prefix into a cache that holds four, evicting something another request was about to hit. The single-request formula ignores both. The next section tests whether that externality is large enough to matter.

What happens if you set the limit at break-even?

I built a small discrete-event simulation. It is illustrative, loosely modeled on the code-generation workload, and it is not a reproduction of llm-d's benchmark. The assumptions:

• 10 endpoints, each a FIFO queue serving prefill at 20,480 tokens/s. Decode and continuous batching are ignored.
• 40 prefix groups of 55,000 tokens, requested with Zipf skew (exponent 1.1, so the most popular group takes 27.2% of traffic and the top five take 58.5%). Each request adds 1,500 new tokens.
• Each endpoint's cache is an LRU holding 4 prefixes, from the 64 GiB budget above.
• Poisson arrivals, 600 s per run, 5 seeds averaged. The router sees exact queue length and cache contents, which a real router only approximates.

Three policies: round-robin; always sticky (the warm endpoint with the least queue, spill only when no endpoint is warm); and sticky until saturated (spill when the best warm endpoint's queue passes tau).

The core of the third policy is short:

warm = [i for i in range(N) if g in cache[i]] least = min(range(N), key=lambda i: pend[i]) # pend = queued tokens if warm: bw = min(warm, key=lambda i: pend[i]) e = bw if pend[bw] <= tau else least # stick until saturated else: e = least 

Three things in that table can be checked by hand, which is how I convinced myself the simulation is not producing noise.

Round-robin's hit rate is 35%, so its average request costs 0.35 x 1,500 + 0.65 x 56,500 = 37,250 tokens. Ten endpoints deliver 204,800 tokens/s, which caps the fleet at 5.5 requests/s. The simulation falls over between 4 and 6. Its huge numbers at 10 req/s and above are the backlog after a 10 minute window, not a steady-state latency.

Always-sticky ties each prefix to one endpoint. The hottest prefix takes 27.2% of traffic at 73 ms per request, so its endpoint saturates at 1 / (0.272 x 0.0732) = 50 req/s. The simulation is healthy at 40 and in trouble at 60. That is hot-spotting, and a saturation release fixes it: at 60 req/s p90 drops from 121.8 s to 3.21 s.

At low rates p90 sits near 2.76 s for both sticky policies. That is one cold fill, and it comes from first-time prefixes and LRU evictions. No router removes it.

The last row is the part I did not expect to write. At 80 req/s the saturating policy does worse than always-sticky (1,000 s against 292 s) because the fleet is simply short of capacity, and spilling turns scarce capacity into cold fills. A saturation release helps a skewed fleet that has total headroom. It does not create headroom.

Now the sweep that answers the break-even question. Same simulation at 60 req/s, varying tau only:

Below roughly 110,000 tokens the system falls into the same state as round-robin. A spill creates a cold fill, the cold fill evicts a warm prefix, the next request for that prefix misses, spills again, and the hit rate sinks to 35% and stays there. Above the cliff everything is fine, and then latency climbs slowly as stickiness turns back into hot-spotting. The shape is a cliff on the low side and a slope on the high side, which argues for erring high.

Be careful with the exact position of the cliff. It comes from my parameters (4 slots, 40 groups, 60 req/s) and a router that sees perfect state. A real fleet has chunked prefill, imprecise cache indexing and a different mix. What transfers is the mechanism and the asymmetry, not the number 110,000.

How do you calibrate this on your own fleet?

1. Measure R_peak where you will run it. Time a prefill of one full scheduler batch on the real model, tensor-parallel layout and engine flags. R_peak is a property of that exact setup, and it will move if you change max-num-batched-tokens.
2. Pick T_max from your TTFT objective, not from the post. Fourteen seconds is the benchmark's tolerance, and an interactive product will want less. Compute tau and express it in scheduler chunks so it is easy to sanity check.
3. Compute the break-even for your median prefix (P tokens, so P / R_peak seconds) and confirm your tau is well above it. If it is not, you are in the region my simulation collapses in.
4. Count how many prefixes fit per endpoint with the KV-bytes formula. If everything fits everywhere, routing barely matters and round-robin is fine.
5. Replay a day of real prefix-hash traffic through a simulator like the one above before touching production, and watch hit rate, not only latency. A falling hit rate under load is the early warning for the cliff.
6. Match the load signal to the bottleneck. Token load for prefill-bound traffic, request count for decode-bound.

What should you not conclude from this?

The 2.9x and 2.0x figures come from the llm-d authors on their own 10-server cluster and workloads. They are a strong reason to test, not a forecast for your traffic. The decode-bound result at parity says the same thing from the other side.

My simulation is a one-queue-per-endpoint toy. It omits decode, continuous batching, cache-indexing error and multi-tenant priorities, all of which the production router handles. It supports one claim only: a stickiness limit set at the single-request break-even can be catastrophic, because the cost of a spill lands on other requests.

FAQ

What is prefix-cache-aware routing?
Sending a request to an inference endpoint that already holds the KV cache for the start of its prompt, so the endpoint can skip recomputing those tokens and only prefill the new ones.

What does "sticky until saturated" mean in llm-d?
Requests stay on the cache-warm endpoint until its queued token load passes a calibrated limit (tau = R_peak x T_max). After that, routing falls back to load alone.

How is llm-d's 286,720-token limit derived?
In the published setup, 8,192 tokens take 0.40 s, so one endpoint prefills 20,480 tokens/s. With a 14 s TTFT degradation tolerance, 20,480 x 14 = 286,720 tokens, which equals 35 batches of 8,192.

Is round-robin ever the right choice?
Yes, when all your prefixes fit on every endpoint, or when requests share little prefix. Then there is no cache to protect, and the decode-bound benchmark at parity shows the gain can vanish.

Why not set the limit at the break-even queue gap?
Because a spill costs more than one request: a cold fill uses 2.76 s of the receiving endpoint's capacity and can evict a warm prefix. In my simulation, limits at or below about twice the break-even collapsed the hit rate to 35%.

Does this apply outside Kubernetes?
The logic does. It needs only per-endpoint queue depth in tokens and some record of which endpoint holds which prefix, which you can approximate by hashing prompt prefixes.

Further reading

• Prefill/Decode Disaggregation in vLLM: The KV Transfer Math That Decides If It Pays Off
• KV Cache Offloading to NVMe: The Break-Even Bandwidth Math and Where It Does Not Pay Off

Sources: llm-d, "Sticky Until Saturated: Token-Aware Routing in llm-d"; llm-d, "llm-d v1.0: State-of-the-art inference, ready for Day 2"; IBM Newsroom, 2026-10-20; Qwen3-32B config.json.