Nginx proxy_buffering Ate My LLM Stream: 0.4s First Token Became 6.1s

On my laptop the chat app typed like a person. Tokens dripped in 0.4 seconds after I hit enter. On the server, the same app sat there with a blinking cursor for six seconds, then dumped the whole answer on screen at once like someone pasting from Notepad.

Same code. Same model. Same prompt. The only new thing between my browser and the app was nginx, and nginx proxy_buffering was quietly collecting my LLM stream into a bucket and pouring it out only when the bucket was full.

I spent two hours on this, measured 200 streams through four configs, and found a second buffer hiding behind the first one. Here is the whole autopsy.

TL;DR

• Nginx buffers proxied responses by default (proxy_buffering on). For Server-Sent Events from an LLM, that means tokens wait in a buffer until it fills or the response ends.
• In my test, 128 of 200 answers were under 8 KB total, so they arrived in one single chunk at the very end. Median time to first byte went from 0.41s (direct) to 6.1s (through nginx).
• proxy_buffering off fixed most of it, but 23 of 200 streams were still bursty because gzip was compressing text/event-stream. The gzip encoder holds small writes too.
• The fix that held: proxy_buffering off, gzip off on the streaming location, and the app sending X-Accel-Buffering: no so the fix survives someone else's nginx config.
• Test with curl -N and a timer on each chunk. "It eventually shows the answer" is not a passing test.

The setup

Nothing fancy:

• A FastAPI endpoint that calls a hosted LLM with streaming on and re-emits each text delta as an SSE event via StreamingResponse with media_type="text/event-stream".
• Uvicorn on port 8000.
• Nginx in front for TLS, on a small VPS. The nginx config was the "sane defaults" block I've been copying between projects for years, which turns out to be the problem.

location /api/ { proxy_pass http://127.0.0.1:8000; proxy_http_version 1.1; proxy_set_header Host $host; } gzip on; gzip_types text/plain text/css application/json text/event-stream; 

Look at that last line. Past me added text/event-stream to gzip_types because "compress all the text." Hold that thought.

How do you measure whether an SSE stream is actually streaming?

You record the arrival time of every chunk on the client and look at the gaps. Watching the browser is useless because a 2-second stall and a 6-second stall both just feel "slow."

I wrote a 30-line script with httpx that opens the stream, timestamps every chunk it reads off the socket, and counts how many SSE events (data: lines) landed in each chunk.

import time, httpx def probe(url, payload): t0 = time.perf_counter() chunks = [] with httpx.stream("POST", url, json=payload, timeout=120) as r: for raw in r.iter_raw(): chunks.append((time.perf_counter() - t0, raw.count(b"data:"))) first = chunks[0][0] bursty = any(n > 5 for _, n in chunks) return first, len(chunks), bursty 

My definition of "bursty": any single network chunk carrying more than 5 SSE events. A healthy token stream sends roughly one event per chunk. A buffered one sends dozens glued together.

I ran 200 prompts, a mix of short questions ("what does git rebase --onto do") and long ones ("write a migration plan for…"), through each config.

Why does nginx buffer LLM streaming responses?

Nginx buffers proxied responses because proxy_buffering is on by default. It reads the upstream response into memory buffers (sized by proxy_buffer_size and proxy_buffers, typically 4 KB or 8 KB pages) and forwards to the client in buffer-sized pieces, so a slow client doesn't hold an upstream connection open.

That default is great for serving a 2 MB JSON payload to a phone on bad Wi-Fi. It's terrible for an LLM that emits 30 bytes every 40 milliseconds.

Here is what the default config did to my 200 streams:

128 answers came back as exactly one chunk. Every one of those was under 8 KB of total SSE output. The buffer never filled, so nginx waited for the upstream to close the response and then sent everything.

The other 72, the long answers, did "stream," but in fat slabs. You'd get nothing for 3 seconds, then a paragraph, then nothing, then a paragraph. Which explains a bug report I'd ignored for weeks: "streaming works on long answers but not short ones." I had assumed the user was confused. The user was right.

Does proxy_buffering off fix SSE streaming in nginx?

Mostly. Setting proxy_buffering off makes nginx forward upstream bytes to the client as soon as they arrive, which dropped my median first byte back to 0.44s. But it did not fix every stream.

location /api/ { proxy_pass http://127.0.0.1:8000; proxy_http_version 1.1; proxy_set_header Host $host; proxy_buffering off; } 

23 streams were still bursty. Not as badly, the worst gap was about 1.2 seconds, but clearly not token-by-token. And they weren't random: every bursty run was from a client that sent Accept-Encoding: gzip. My probe script did by default. Browsers always do.

Why does gzip break Server-Sent Events?

Gzip breaks SSE because a compressor works best on large blocks, so it accumulates small writes before emitting compressed output. Nginx's gzip filter only flushes on its own buffer boundaries or at the end of the response, so 30-byte token events sit inside the compressor waiting for company.

Remember the gzip_types text/event-stream line I was proud of? That was buffer number two. Turning off proxy buffering removed the first bucket, but the gzip filter was a second, smaller bucket right behind it.

Why only 23 of 200 and not all of them? The bursty ones were the fast streams. When the model emitted tokens quickly, several events landed inside the compressor before it flushed. Slower streams happened to flush closer to one event per chunk. So the bug was worse exactly when the model was performing best, which is a special kind of insulting.

The fix is to never compress the stream:

location /api/stream { proxy_pass http://127.0.0.1:8000; proxy_http_version 1.1; proxy_set_header Host $host; proxy_set_header Connection ""; proxy_buffering off; proxy_cache off; gzip off; } 

And I took text/event-stream out of the global gzip_types. Compressing tiny token events saves almost nothing anyway. The JSON wrapper around a token is already most of the bytes.

Zero bursty streams. The server now types like my laptop.

How do you make the fix survive someone else's nginx config?

Send X-Accel-Buffering: no from the application. Nginx reads that response header from the upstream and disables proxy buffering for that one response, regardless of what proxy_buffering is set to in the config.

return StreamingResponse( event_gen(), media_type="text/event-stream", headers={ "X-Accel-Buffering": "no", "Cache-Control": "no-cache", }, ) 

This matters because the nginx config is often not yours. A teammate refactors it. A hosting platform puts its own nginx in front. A "performance hardening" PR turns buffering back on globally. The header travels with the response, so the stream stays unbuffered even when the config drifts.

Two caveats from my own testing:

1. The header does not turn off gzip. I re-enabled gzip with the header set, and the bursty streams came back. You still need gzip off (or text/event-stream out of gzip_types) on the proxy.
2. Other proxies have their own buffers. A CDN or a second load balancer in front of nginx can buffer independently. Run the probe against the public URL, not just against nginx on localhost.

The checklist I now run on every streaming endpoint

• curl -N -H "Accept-Encoding: gzip" https://your-host/api/stream and watch whether text appears token by token. The -N disables curl's own buffering, and the gzip header reproduces what browsers send.
• Run a chunk-timing probe on at least 50 prompts that include short answers. Short answers are where buffering hides, because they never fill a buffer.
• Grep your nginx config for gzip_types and make sure text/event-stream isn't in it.
• Set X-Accel-Buffering: no in the app anyway. It costs one line.
• Measure first-byte time at the client, not in your app logs. My app logs said 0.4s the entire time. The app was telling the truth. The bytes were just sitting in nginx.

The honest cost of this bug: two hours of debugging, one dismissed bug report I owe an apology for, and roughly six months of users thinking the app was slow when the model was fast.

So, why is my LLM stream slow behind nginx?

Your LLM stream is slow behind nginx because proxy_buffering is on by default, so nginx collects SSE token events into 4-8 KB buffers and only forwards them when a buffer fills or the response ends. In my test that made 128 of 200 short answers arrive as a single chunk and raised median first byte from 0.41s to 6.1s. Set proxy_buffering off on the streaming location, also set gzip off there (gzip holds small writes too, which left 23 of 200 streams bursty), and send X-Accel-Buffering: no from your app so the fix survives config changes. Then verify with curl -N and per-chunk timing, using short prompts.

Written by the developer behind Preterview, an interview prep platform.