Salesforce Fixed Its Model-Serving Capacity Problem by Admitting It Couldn't See It
Salesforce serves models across 30+ Regions and hundreds of AWS accounts — and its own gateway couldn't tell when a Region was out of capacity until requests started failing. The fix isn't more GPUs. It's a router that finally knows where the room is.
That's the honest version of this story: cross-Region inference routing in Amazon SageMaker AI is a visibility and plumbing upgrade, not a magic capacity multiplier. And that's exactly why it's worth paying attention to.
The problem was fragmentation, not shortage
A Region could run out of model-serving capacity while instances elsewhere sat idle. Salesforce provisions around each customer's traffic, and customer peak windows overlap — provisioning every Region for its own worst peak is expensive, and onboarding a new business unit at short notice makes the fleet grow in lumps. The result: some locations hoarded unused instances while others exhausted theirs.
Salesforce already had a cross-Region fallback. It was, by the account's own admission, flying blind: basic health checks and retries, no capacity signals from either end, so the gateway couldn't reliably judge whether a destination actually had room. Worse, the old path left the VPC through a NAT gateway and crossed the public internet to get there.
What the router actually does
- One invocation path, many destinations. Applications call a router by name with a routing target — a logical model, like a toxicity classifier — and the router picks among eligible endpoints across Regions and accounts. It uses Salesforce's existing deployments; it deploys nothing, copies no artifacts, adds no instances.
- Local first, failover with a conscience. Requests stay local while the local endpoint serves. When platform faults, 5xx errors, throttles, or timeouts pile up, traffic shifts gradually — typically starting within about a minute — and returns more slowly after recovery, to avoid flapping.
- It reacts to failure, not to load. This is the qualification that matters: the documented mechanism moves traffic on request failures. Scaling signals alone don't move traffic, latency doesn't trigger failover, and a slow response might just be a long prompt. Don't read this as "always picks the fastest Region" — it doesn't.
- Residency stays Salesforce's job. Data-residency checks remain in the Internal Gateway, pinning tagged requests to in-Region targets. The router never gets to overrule that.
The real win is the receipt
Every routed request now logs two fields: InvokedRoutedResourceArn (where it ran) and RoutingReason — local-preferred, failover, capacity-available, or only-healthy. Combined with OpenTelemetry metrics that split destination time from routing overhead, Salesforce can finally answer "why did this request go to Ohio?" per request, not per incident review.
That's the part vendors usually skip: the telemetry to prove whether the routing actually helped. The source is candid that no fleet-wide improvement in availability, utilization, or cost is measured yet — the walkthrough confirms routing across account and Region boundaries, not production behavior at scale. The dashboard from AWS's failover test is a test model, not a Salesforce production benchmark.
The take
Cross-Region routing is the right architecture for a capacity-fragmentation problem — and Salesforce's own history proves the old DIY fallback was the wrong tool for it. But routing is not provisioning: it can only pick among destinations that already exist for that model. The honest scorecard is still open, and now, at least, there's telemetry to fill it in.
Go deeper
Read the full analysis on Agentic Affairs: How Salesforce uses cross-Region inference routing to address model-serving capacity gaps
Originally published on Agentic Affairs (https://agentic-affairs.com)

