Most teams measure an inference call from the moment the request leaves the application to the moment the model answers. That single number hides where the time actually went. In a multi-cloud estate, the model is often the fastest part of the journey. The rest is spent crossing clouds: resolving a private name, opening a connection over an interconnect, fetching context from a vector store in another cloud, calling a reranker, and only then sending the assembled prompt to the model.

If the whole chain has a budget, say 800 milliseconds for an interactive feature, the network does not get whatever is left over. It has to be budgeted first, line by line, or the feature will feel slow in ways no model upgrade can fix.

Write the budget down, hop by hop

Start with the user-facing target and work backwards. A practical budget for one retrieval-augmented request might look like this on paper:

  • DNS and connection setup to private endpoints: 30 ms
  • Retrieval query to the vector store, including the return of the top results: 120 ms
  • Fetch of source documents or metadata for those results: 80 ms
  • Reranking or filtering call, if one exists: 60 ms
  • Prompt assembly and the call to the model endpoint: the remainder
  • Response streaming back to the caller: counted from first token, not from completion

The exact numbers matter less than the habit. Once each hop has a number, a slow request stops being a mystery. Timings per hop show whether the vector store got slower, the cross-cloud path got longer, or the model itself is the problem. Without that split, every incident starts with the same argument about which team owns the slowness.

Measure first-token latency as well as total latency. Users judge a streaming feature by how quickly it starts answering. A path that adds 300 ms before the first token feels broken even if the total time looks acceptable in a dashboard average.

Count the crossings, not just the distance

Latency between clouds is not only about geography. Each crossing adds connection setup, TLS negotiation, and often a pass through a NAT device, a firewall, or an inspection point. A design that crosses between clouds four times inside one user request can be slower than a design that crosses once, even when both use the same regions.

Map one real request end to end and count the crossings. Common surprises:

  • The application lives in one cloud, embeds the question using a model in a second cloud, then queries a vector store back in the first cloud. That is two crossings before retrieval has even finished.
  • Retrieval returns document IDs, and the application then fetches each document with a separate call across the link. Ten small fetches can cost more than the search itself.
  • A logging or policy check calls a central service in another region on every hop, quietly adding its round trip to every stage.

Reducing crossings usually beats shaving milliseconds off a single link. Batch document fetches into one call. Co-locate the embedding step with the store it queries most. Cache stable context, such as product documentation or policy text that changes rarely, close to the application, and invalidate it on publish rather than fetching it fresh on every request.

Place the vector store by query pattern, not by team

Vector stores tend to land wherever the data team first built them. Inference traffic then pays for that history on every request. Placement should follow the query pattern instead:

  • If most queries come from applications in one cloud, the primary store belongs in that cloud, with replication to the others for resilience, not the other way round.
  • If two clouds both serve heavy inference traffic, consider two readable replicas and route each request to the nearer one. Accept that results can differ briefly after an update, and make that lag visible rather than pretending it does not exist.
  • If the corpus is small and changes rarely, a scheduled copy into each serving cloud can be simpler and faster than live cross-cloud queries on every request.

Whichever pattern fits, keep the path private. Retrieval traffic carries user questions and document fragments, which are often the most sensitive text in the whole flow. It should travel over private interconnects or VPNs to private endpoints, with the same identity and policy checks as any other service-to-service call, and it should never fall back silently to a public endpoint when the private route fails. A silent fallback trades a visible outage for an invisible exposure.

Connection setup is part of the budget

Interactive inference creates many short connections, and short connections pay full setup costs every time. DNS lookups, TCP handshakes, and TLS negotiation across a high-latency link can together exceed the retrieval query itself.

Three fixes cover most of the waste. Reuse connections with pooling and keep-alive tuned longer than the gap between typical requests, and check that idle timeouts on the path do not close pooled connections early. Resolve private names through a resolver that caches sensibly and is reachable without crossing clouds. Pre-warm a small pool of connections to the model endpoint and the vector store at startup and after deploys, so the first real users do not pay the cold-start cost that load tests never see.

None of this needs exotic tooling. It needs someone to measure setup time separately from query time, which most tracing setups can do once the spans are named honestly.

Degrade in order, on purpose

When the budget is blown, something has to give. Decide the order in advance, while nobody is on a bridge call:

  1. Reduce the number of retrieved documents, for example from ten to five, and note the change in the response metadata.
  2. Skip the reranking step and use the raw retrieval order.
  3. Serve from the nearer replica or the cache, even if it is minutes stale, and mark the answer as based on cached context.
  4. Only then consider failing the request or queuing it.

Each step trades a little answer quality for a lot of latency, and each is reversible when the path recovers. What does not work is the default behaviour of most systems: retry the slow hop, add its timeout to the total, and return an error after the user has already waited several seconds. Retries across clouds should be rare, short, and counted against the same budget, not added on top of it.

A short checklist to start

  • Write a per-hop latency budget for one inference feature and instrument each hop separately.
  • Trace one real request and count how many times it crosses between clouds. Remove at least one crossing.
  • Place or replicate the vector store according to where queries originate, and keep retrieval on private paths with no silent public fallback.
  • Measure connection setup separately, pool connections, and pre-warm them after deploys.
  • Define the degradation order in writing and test it by slowing the cross-cloud path in staging.

Teams that want a structured way to map these inference paths, place retrieval and model endpoints consistently across AWS, Azure, Google Cloud, and Oracle estates, and keep the traffic private by default can look at the approach set out at CloudVerg, where VergMind and VergMesh are used for exactly this kind of AI connectivity work. The checklist above stands on its own either way. Start with the one feature users say feels slow, write its budget down, and let the numbers show which hop is spending it.