TensorRT-LLM 1.4 Makes trtllm-serve Run 8 HTTP Frontends by Default: RAM Cost, Connection Balance, and What Still Differs

TL;DR

TensorRT-LLM 1.4.0rc3 (GitHub release 2026-11-06) flips one default that changes the shape of your deployment: trtllm-serve now starts 8 HTTP frontend processes instead of 1. The release notes mark it BREAKING (PR #19379). I read that PR, the earlier prototype PR #16523, the metrics fix #19394, the current docs on main, and the release notes, then ran two small simulations of my own. I did not run a GPU workload.

• The reason is host-side, not GPU-side. The prototype PR measured gen_preprocessing p50 at 54 ms (concurrency 512), 109 ms (1024) and 1,520 ms (2048) on a DeepSeek V4 disaggregated generation worker while decode step time stayed flat. One asyncio loop was the queue.
• It costs host RAM. The PR measured about 1.99 GB resident per attached frontend and about 2.5 GB for the launcher on a Qwen3-0.6B test. That is roughly 16.4 GB for 8 frontends versus 2.5 GB for 1, and time to healthy went from 54 s to 154 s in that one test. GPU memory is unchanged.
• Balance is per connection, not per request. In my model, 8 frontends sharing a load worth 3 event loops stay comfortable only with many connections. With 16 long-lived connections the busiest frontend is above 1.0 utilization in 7.7% of random draws.
• Some of the scary caveats in the PR body are already fixed. The metrics sharding described in #19379 was addressed by #19394, merged two days earlier and shipped in the same rc. Still different: the stateful Responses API store is off, /health is per frontend, and perf-metrics files are per frontend.
• Where this does not apply. PyPI's latest non-prerelease tensorrt-llm is 1.2.1 as I write this, so a plain pip install does not get this default.

What exactly changed in trtllm-serve 1.4.0rc3?

The CLI default for --num_serve_frontends on the trtllm-serve serve subcommand went from 1 to 8. The launcher process owns the engine. The other 7 frontends re-exec the command line, attach to the same executor, and bind the same port with SO_REUSEPORT, so clients still see one URL.

Three details from the PR are easy to miss:

1. Only the CLI default changed. LlmArgs.num_serve_frontends still defaults to 1, so trtllm-bench, trtllm-eval and a bare Python LLM() do not pay for the multi-frontend topology.
2. YAML wins over the new default. num_serve_frontends: 1 in the --config file overrides it, as does --num_serve_frontends 1 on the command line.
3. Some configurations quietly fall back to 1. If you did not type the flag and the config cannot run several frontends, the server logs why and runs one. The current docs list --port 0 or --report_addr, orchestrator_type set to rpc or ray, enable_resource_governor, and --enable_sleep_mode. --grpc also keeps one unless you ask for more. If you type --num_serve_frontends 8 together with one of those, startup fails instead.

The same release candidate carries two other BREAKING entries (#21036 removes unused LlmRequest fields, #21041 removes obsolete TensorRT-backend compatibility surfaces). Those are API removals. The frontend default is the one that changes what a plain launch does on your host.

Here is the shortest way to opt out:

# CLI trtllm-serve <model> --num_serve_frontends 1 # or in the --config YAML num_serve_frontends: 1 

Why would one HTTP process become the bottleneck?

Every request on a single trtllm-serve process goes through one asyncio event loop: json.loads, pydantic validation, and then every streamed chunk written back. The prototype PR (#16523, merged 2026-07-22) reported this on a DeepSeek V4 disaggregated generation worker:

Doubling concurrency from 512 to 1,024 roughly doubled the delay. Doubling it again multiplied it by about 14, while decode step time stayed flat. The PR author called this an M/M/1-style blow-up, and I agree with the shape. In a single-server queue the mean wait grows like utilization divided by one minus utilization, so it looks harmless until the loop approaches saturation and then explodes. Adding GPU capacity does nothing for it, because the GPU is not the queue.

That is why eight processes help: the same parsing and streaming work is spread over eight event loops and eight GILs. The PR says the setting was already the production configuration in the authors' agentic disaggregated setups, with 8 on context workers and 4 on generation workers.

Does SO_REUSEPORT really balance the load?

The kernel assigns each new connection to one of the listening sockets, effectively by hashing the connection tuple. The docs say it plainly: each request is handled entirely by the frontend that accepted its connection. Once a client holds a keep-alive connection, all of its requests land on the same frontend.

So the question that matters is how many long-lived connections your clients open. I modeled it with the standard library only. The assumptions are mine and deliberately simple: connections are assigned uniformly at random to 8 frontends, every connection carries equal load, and the total offered load equals 3 single-process event loops (so one frontend alone would be hopelessly saturated). Perfect balance would put each frontend at 0.375 utilization, a mean M/M/1 wait of 0.6 service times.

The core of the model:

import random K, RHO_TOTAL = 8, 3.0 def busiest_utilization(n_conn): counts = [0] * K for _ in range(n_conn): counts[random.randrange(K)] += 1 return RHO_TOTAL * max(counts) / n_conn def wait_factor(u): # M/M/1 mean wait in service times return float("inf") if u >= 1 else u / (1 - u) 

I ran 20,000 draws per connection count with a fixed seed:

With 1,024 connections you are within 13% of perfect balance. With 16 you are at a median utilization of 0.75, a mean wait of 3.0 service times against the balanced 0.6, and about one draw in thirteen saturates a frontend. This is an illustration, not a measurement of TensorRT-LLM. Real traffic is lumpier than equal-weight connections (a long streaming response weighs more than a short one), which I expect makes the tail worse, but I have not measured that. If your traffic comes through a router or gateway with a small connection pool, count its connections to this server before assuming 8 frontends means 8x headroom.

What does the new default cost in host memory and startup time?

The PR's verification section is the only measurement I found, so I use it as is. One container with 1x GB300, Qwen3-0.6B, launched the way that cluster runs production:

Resident memory was about 1.99 GB per attached frontend and about 2.5 GB for the launcher. Every extra frontend is a full Python process that imports TensorRT-LLM and loads the tokenizer, which is what the docs warn about. The arithmetic for the default is 2.5 + 7 x 1.99 = 16.43 GB, about 13.9 GB more than a single frontend.

Two things to keep in mind. First, this is one run on a tiny model, so treat 1.99 GB as an order of magnitude, since a larger tokenizer or extra imports will move it. Second, a container memory limit sized around a single serving process will now get OOM-killed during startup. Attached frontends hold no GPU memory: in the PR, nvidia-smi listed only the worker (58.7 GiB including KV cache) and the launcher (748 MiB), identical with 1 or 8 frontends.

Is the metrics sharding in the PR body still a problem?

This is the part I almost got wrong. The body of #19379 lists a caveat: /metrics, /prometheus/metrics and /health are answered by whichever frontend accepted the connection, so scrapes see one frontend's share and counters appear to reset. It cites a measurement of trtllm_request_success_total showing 5 distinct values (6, 7, 8 and 10 among them) for 64 requests across 32 scrapes.

But #19394, "make multi-frontend metrics, profiling and liveness launcher-owned", was merged on 2026-10-28, two days before the default flip, and I confirmed with the compare API that both land first in rc3 (neither is in rc2). It reports the same 32 scrapes returning one identical value (64) afterwards. The mechanism: the launcher sets one shared PROMETHEUS_MULTIPROC_DIR before it spawns children, iteration-derived series come from the launcher only, and attached frontends forward GET /metrics to the launcher over a Unix socket. The current docs on main match that: /metrics is forwarded and /prometheus/metrics covers every frontend.

Why does this matter to you? Because Prometheus treats any decrease in a counter as a reset. To show what the sharded behavior would have done to a dashboard, I simulated 64 requests landing on 8 per-process counters, scraped at random every second request:

With shared counters the increase is the true value, 62 between the first and last scrape, every time. With per-frontend counters the 5th to 95th percentile spread is 47 to 86 (minimum 23, maximum 111), and 38% of runs were off by more than 10. Illustrative again, but it is why a build without #19394 and with 8 frontends is a bad combination for anything alerting on request rates.

You can check your own server in a few seconds. This probe scrapes a counter repeatedly and flags any decrease. I tested it against a mock HTTP server that I wrote in both modes (it reported consistent for shared counters and a drop count for sharded ones). I have not pointed it at a real TensorRT-LLM server.

import re, time, urllib.request def total(url, name): text = urllib.request.urlopen(url, timeout=10).read().decode() return sum(float(m.group(1)) for m in re.finditer( r'^' + re.escape(name) + r'(?:\{[^}]*\})?\s+([0-9.eE+-]+)', text, re.M)) def probe(url, name="trtllm_request_success_total", n=30, gap=0.05): s = [] for _ in range(n): s.append(total(url, name)); time.sleep(gap) drops = sum(b < a for a, b in zip(s, s[1:])) return {"distinct": len(set(s)), "drops": drops, "verdict": "per-frontend counters" if drops else "consistent"} 

Send some traffic while it runs. If the counter is flat you cannot tell the difference.

What still behaves differently with several frontends?

Per the current docs and the PRs, these remain:

The /health row is the one I would test in your environment. A load balancer health check that lands on a different frontend each time is exactly the kind of thing that passes in staging and flaps in production.

What should I check before upgrading?

1. Are you on 1.4.0rc3 or main, not 1.2.1? If you install from PyPI without --pre, this does not affect you yet.
2. Does the memory limit cover 16 GB of extra frontends, or should you set num_serve_frontends to 2 or 4? The PR gives no sizing rule, only that production disaggregated configs use 8 and 4.
3. How many connections do your clients hold? Under roughly 32, expect imbalance (table above).
4. Do you use the Responses API store? If yes, pin 1.
5. Do your dashboards and alerts survive? Run the probe, and confirm your build includes #19394.
6. Did you rely on a quiet fallback? If your config hit one of the fallback conditions, you are already on 1 frontend and will see a log line saying why.
7. Re-measure. Record gen_preprocessing p50 and time to healthy before and after, since the numbers in the PR come from one cluster and one small model.

What I did not do

I ran no GPU workload and no TensorRT-LLM server. The latency, memory and startup numbers are quoted from the PRs, and I attribute them as such. The two simulations are my own, use simplifying assumptions stated above, and should be read as intuition about the mechanism, not as predictions for your fleet. My statements about which rc contains which PR come from the GitHub compare API and the rc3 release notes on 2026-11-11.

FAQ

Does running 8 frontends use more GPU memory?
No. In the PR's test, attached frontends held no GPU memory. nvidia-smi listed only the worker and the launcher, in both the 8-frontend and 1-frontend runs.

How do I go back to a single frontend?
Pass --num_serve_frontends 1 or set num_serve_frontends: 1 in the --config YAML. The YAML value overrides the new CLI default.

Will pip install tensorrt-llm give me this default?
Not today. The latest non-prerelease on PyPI is 1.2.1. The change is in the 1.4.0rc3 release candidate (also uploaded to PyPI on 2026-11-06) and on main.

How many frontends should I run?
The PR does not give a formula. Its production configs use 8 on context workers and 4 on generation workers. I would start from the free CPU cores and the RAM budget above, then measure request-handling latency at your real concurrency.

Is the stateful Responses API store gone?
It is disabled whenever more than one frontend runs, because it lives in each process's memory. With --num_serve_frontends 1 it works as before.

Can another process steal traffic through SO_REUSEPORT?
The docs say the launcher refuses a port that another server already uses, which is the guard against a second server of the same user joining the port.

Further reading

• Day-0 Model Support Is a Fork: What 26 NVIDIA Dynamo Snapshot Builds Say About Pinning Your LLM Serving Stack
• Your LLM Serving Engine Ships Every 13 Days: Reading vLLM's Breaking-Change Lists as Upgrade Risk Data

Sources

• TensorRT-LLM v1.4.0rc3 release notes
• PR #19379: default num_serve_frontends to 8 with single-frontend fallback
• PR #16523: multi-process HTTP frontends on the classic IPC executor path
• PR #19394: multi-frontend metrics, profiling and liveness launcher-owned
• trtllm-serve docs, Multiple HTTP Frontends
• tensorrt-llm on PyPI