If you call a model hosted in mainland China from a server in Europe, the Americas or Southeast Asia, you have probably seen it: the first token takes a while, some requests hang for no clear reason, and the evening hours (yours or Beijing's) are worse than the morning. The model is usually fine. The problem is the path between you and it, plus a client that was never tuned for a long, lossy route.

This is the checklist I use. Everything here is vendor-neutral and works with any OpenAI-compatible /chat/completions endpoint. The full scripts are at https://apigotonotes.wordpress.com/2026/10/20/calling-chinese-llms-from-outside-china-measure-first-then-fix-latency-and-stability/; they are tested against a local mock server (included). Endpoint and key come from environment variables (LLM_BASE_URL, LLM_API_KEY); MODEL_ID is a placeholder for whatever model you use.

1. Why cross-border routes jitter

A request from, say, Frankfurt to a data center in China crosses a small number of international links and peering points. At busy hours those links get congested, packets get dropped, and TCP retransmits. Each retransmit costs at least one round trip, and on a long route a round trip is expensive. Routes can also change mid-day, so the same request can behave differently an hour later.

Long round trips also make new connections costly. Before the first byte of your request is sent, a fresh HTTPS connection needs a TCP handshake (one round trip) and a TLS handshake (one more with TLS 1.3, two with TLS 1.2). If your code opens a new connection per request, you pay that every time.

So there are two separate problems: latency (round trips, handshakes, time to first token) and stability (loss, stalls, resets). The fixes overlap, but you should measure them separately.

2. Measure before you change anything

Don't trust a single request, and don't trust numbers from someone else's network, including mine. Measure from the servers your users actually hit, at the hours they actually use them.

Quick per-phase view with curl (every run is a new connection, so this shows the cold path):

curl -sS -N -o /dev/null \ -H "Authorization: Bearer $LLM_API_KEY" -H "Content-Type: application/json" \ -d '{"model":"MODEL_ID","stream":true,"max_tokens":32,"messages":[{"role":"user","content":"Say hi"}]}' \ -w 'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n' \ "$LLM_BASE_URL/chat/completions"

ttfb here is the first byte, which with streaming is often just the headers. What users feel is time to first token (TTFT). The Python script measure_ttft.py records TTFB, TTFT and total time for N runs per base URL. It reuses one connection, so it reports the cold first run separately from the warm runs and prints p50/p90/max:

python measure_ttft.py -n 20 --model MODEL_ID \ --base-url "$ROUTE_A" --base-url "$ROUTE_B"

Run it a few times a day for a couple of days. The spread (p90 vs p50) tells you more about stability than the median does.

3. Stream, and read the stream to the end

Streaming (SSE, "stream": true) doesn't make generation faster. It makes the wait feel shorter, because users see text at TTFT instead of at total time. It also lets you detect a stalled response early.

Minimal parsing with httpx:

with client.stream("POST", f"{base}/chat/completions", headers=h, json=payload) as r: for line in r.iter_lines(): if not line.startswith("data:"): continue # comments / keep-alive pings  data = line[5:].strip() if data == "[DONE]": continue # keep reading to EOF  choices = json.loads(data).get("choices") or [] if choices and (piece := (choices[0].get("delta") or {}).get("content")): yield piece

One thing my tests caught: if you break on [DONE] instead of draining to the end of the response, the HTTP library may close the connection rather than return it to the pool. Then every request pays the handshake again.

4. Split your timeouts

One big timeout hides the problem. Use separate values:

  • connect: short (a few seconds). If TCP can't connect quickly, retrying or failing over beats waiting.
  • read: for streaming, this is the longest silence between two chunks, not the total time. Set it from your own TTFT p99 plus headroom.
  • overall deadline: a hard cap per call, retries included, so a slow trickle can't run forever.

Check your SDK's defaults. The OpenAI Python SDK README, for example, documents a 10-minute default timeout and 2 automatic retries. That's reasonable for batch jobs and far too patient for an interactive app.

5. Retry only what's worth retrying

Retry on 429, 5xx, timeouts and connection errors. Don't retry 400/401/403/404/422: the same request will fail the same way.

  • Use exponential backoff with full jitter: sleep(random.uniform(0, min(cap, base * 2**attempt))). Jitter stops all your workers from retrying in lockstep.
  • Honor Retry-After (seconds or an HTTP date). If it asks you to wait longer than your budget allows, fail over instead.
  • Never retry silently after tokens have streamed. The user has already seen half an answer. Raise an error and let the caller decide.
  • Don't stack retries. If your SDK retries twice and your wrapper retries three times, one bad minute becomes up to 12 requests (4 × 3). Pick one layer and set max_retries=0 on the other.

6. Reuse connections

Create one HTTP client (or SDK client) per process and share it. Keep-alive skips the TCP+TLS handshake on every request after the first, and on a long route that's the cheapest latency win you'll get. In the mock test suite, four calls on a shared client used one TCP connection. Three calls with a new client each time used three.

7. Choose the route deliberately

You have roughly two options:

  1. Direct: your server talks to the provider's endpoint in China over the public internet. It's the simplest path with no extra party, but you get whatever the cross-border route does that hour.
  2. A gateway near you: you connect to a nearby server (short, cheap handshake), and the gateway carries traffic to China over a link it manages. You add a hop and a third party that sees your traffic, so check its logging and data policy. In exchange, the long leg is no longer the open public route.

Neither one wins by default. Run the measurement script against both from your production region, compare p50 and p90 over a few days, and keep the other route as a fallback.

8. Fallback, health checks and a circuit breaker

Configure at least two endpoints (different base URLs and/or models). Try them in order:

  • A circuit breaker per endpoint: after N consecutive failures, stop sending traffic for a cooldown. Then let exactly one probe through (half-open). Success closes it, failure re-opens it.
  • A health check (GET /models with a short timeout) from a background job, not on the request path.
  • Fail over on retryable errors and on 401/403/404 for that endpoint, but not on 400/422, because a malformed request will fail everywhere.

resilient_llm.py implements all of this in about 300 lines with httpx. The tests cover 429 + Retry-After, 503 backoff, read timeouts, mid-stream drops, connection refused, breaker open/half-open, and keep-alive.

9. Send less, ask for less

Every token you send has to cross the route, and every token you ask for extends total time.

  • Trim conversation history. Summarize old turns instead of resending them.
  • Set max_tokens to what the UI actually needs.
  • Cache answers to repeated prompts on your side.

10. Keep your own latency numbers

Provider dashboards show their side. Log TTFB, TTFT, total time, status and endpoint for every call, then look at p50/p90/p99 per endpoint per hour. The LatencyLog class in resilient_llm.py does this in memory. In production, send the numbers to whatever metrics system you already use. That's how you notice that one route gets worse every evening before your users tell you.

Checklist

  • [ ] Measured TTFT and total (p50/p90) from production, at peak hours, for each route
  • [ ] Streaming on, response drained to EOF
  • [ ] connect / read / overall-deadline timeouts set from your own data
  • [ ] Retries only on 429/5xx/timeouts, with jitter and Retry-After, in one layer only
  • [ ] One shared client per process
  • [ ] Second endpoint + circuit breaker + background health check
  • [ ] Prompt trimming and max_tokens limits
  • [ ] Latency percentiles logged per endpoint

Disclosure: I work on the APIGOTO team (independent of OpenAI, Anthropic and Google). Nothing above depends on any particular gateway or provider; the scripts work with any OpenAI-compatible base URL.

This article was drafted with the help of AI. Code samples are meant to be run against the included mock server or your own OpenAI-compatible endpoint; please verify SDK defaults against the current SDK docs before relying on them.