The first time a Cloud Run region had a bad afternoon, my service was not down for long. It was down long enough. A readiness check that customers rely on started timing out, the status page stayed green because my health endpoint only checked the process, and I spent the outage learning that "serverless" and "highly available" are not the same sentence. Cloud Run scales to zero and absorbs traffic spikes beautifully. It does not, by itself, survive a regional incident. That is a design decision you have to make on purpose, and this article is the design I ended up with: an active-passive multi-region setup for a small FastAPI security service, built by one person, with honest numbers about what it costs and what it does not protect against.
The shape of the setup
There are two regions. One is active and serves all traffic. The other is passive: it runs the same revision, with minimum instances set to zero, and it receives nothing until the load balancer decides the active region is unhealthy. In front of both sits a global external Application Load Balancer with a serverless network endpoint group (NEG) per region. Behind each Cloud Run service is a database the passive region can actually use — and that last sentence is where most of the real work lives.
┌──────────────────────────────┐ │ Global external App LB │ │ + Cloud Armor policy │ └──────┬───────────────┬───────┘ │ │ ┌────────────┴────┐ ┌───────┴─────────┐ │ Serverless NEG │ │ Serverless NEG │ │ us-east1 (active)│ │ us-central1 (passive)│ └────────┬────────┘ └────────┬────────┘ │ │ ┌────────┴────────┐ ┌────────┴────────┐ │ Cloud Run │ │ Cloud Run │ │ (min-instances │ │ (min-instances │ │ = 1, active) │ │ = 0, standby) │ └────────┬────────┘ └────────┬────────┘ │ │ ┌────────┴────────┐ ┌────────┴────────┐ │ Cloud SQL │ │ Cloud SQL │ │ (primary) │ │ (cross-region │ │ │ │ read replica) │ └─────────────────┘ └─────────────────┘ Everything else in this article is a consequence of that diagram: how the load balancer decides a region is dead, how the passive region gets a database it can trust, and how you keep the whole thing cheap enough to leave running.
Step 1: the load balancer has to decide, not you
The load balancer routes to a backend service with two backends — one NEG per region — and it fails over based on health checks. The health check is the most important line in this setup, because a bad one either fails over too late or flaps between regions forever. My health endpoint does not return 200 because the process is alive. It checks the things a request actually needs:
@app.get("/healthz") def healthz(db: Session = Depends(get_db)): # Fail the check if the database is unreachable or # if replication lag is past the threshold we accept. db.execute(text("SELECT 1")) lag = replication_lag_seconds(db) if lag > MAX_ACCEPTABLE_LAG_SECONDS: # 30 seconds, tuned below raise HTTPException(status_code=503, detail="replication lag too high") return {"status": "ok", "region": os.environ["REGION"]} Two decisions are embedded here. First, the check depends on the database, not just the process — a service that cannot reach its database is not a healthy backend. Second, the passive region fails its own health check while replication lag is high, which keeps the load balancer from sending traffic to a region whose data is too stale to serve. Thirty seconds is my threshold, not a universal constant: it comes from the scoring workload, where evidence a minute old is still useful, but evidence five minutes old would produce a score I would not stand behind.
The backend service itself is configured with a failover policy — the passive NEG only receives traffic when every instance in the active NEG is unhealthy:
# Create the serverless NEGs (one per region) gcloud compute network-endpoint-groups create neg-east \ --region=us-east1 --network-endpoint-type=serverless \ --cloud-run-service=scoring-service gcloud compute network-endpoint-groups create neg-central \ --region=us-central1 --network-endpoint-type=serverless \ --cloud-run-service=scoring-service # Backend service with the passive NEG as the failover target gcloud compute backend-services create scoring-backend \ --global --load-balancing-scheme=EXTERNAL_MANAGED --protocol=HTTPS gcloud compute backend-services add-backend scoring-backend --global \ --network-endpoint-group=neg-east --network-endpoint-group-region=us-east1 gcloud compute backend-services add-backend scoring-backend --global \ --network-endpoint-group=neg-central --network-endpoint-group-region=us-central1 \ --failover The --failover flag is the whole point: the passive region gets zero traffic while the active region is healthy. That keeps the cost story simple and, more importantly, it keeps the passive region's behaviour predictable — it is a standby, not a second active that you have to keep in sync under load.
Step 2: the database is the hard part
Compute failover is easy. Data failover is where this design earns its keep, and where you have to be honest about what you are getting.
For Cloud SQL, I run a cross-region read replica in the passive region. The replica is asynchronous, which means there is replication lag — usually a second or two, sometimes more under write bursts — and that lag is the RPO of the whole design. If the active region dies mid-write, the passive region may serve data a few seconds behind. For a readiness score that recomputes from stored evidence, a few seconds of staleness is acceptable. For a payment ledger, it would not be. Name your RPO before you choose your replication, not after.
# Cross-region read replica in the passive region gcloud sql instances create scoring-replica \ --master-instance-name=scoring-primary \ --region=us-central1 \ --tier=db-custom-1-3840 The replica is read-only, which creates a real constraint: the passive Cloud Run service cannot write to it. During a failover, the service runs in a degraded mode — it serves reads and computed scores from replicated data, and it queues writes instead of performing them. That queue is a Pub/Sub topic, and the writes replay once the primary is back or the replica is promoted. This is a deliberate trade-off: I would rather serve a slightly stale score with a clear "data as of" timestamp than refuse to serve anything, and I would rather queue a write than silently drop it. The API response says so explicitly:
@app.get("/readiness/{customer_id}") def readiness(customer_id: str): score = compute_score(customer_id) return { "score": score.value, "status": score.status, # READY TODAY / ACTION NEEDED / ... "evidence_as_of": score.evidence_as_of.isoformat(), "degraded": is_failover_mode(), # True when serving from the replica } The degraded flag and the evidence_as_of timestamp are not cosmetic. They are the contract with whoever consumes the score: during a failover, the consumer knows the data is replicated and how old it is, and can decide for itself whether that is good enough. A failover that silently serves stale data as if it were fresh is worse than an outage — at least an outage is honest.
Firestore, which I use for customer-facing configuration and auth-adjacent data, is the easy half of this story: it is multi-region by design, so the passive region reads the same data with no replication work from me. Put data where the platform already solves the problem, and spend your failover budget only on the store that needs it.
Step 3: keep the passive region cheap and the failover tested
The passive region costs almost nothing while it waits, and that is by design:
- Cloud Run min-instances = 0 in the passive region. Cold start on failover is the price — the first request after a failover takes a few seconds while an instance spins up. I accept that because the alternative, a warm standby, costs a running instance 24/7 for an event that may never happen. If your recovery-time objective cannot tolerate a cold start, set min-instances to 1 and pay for it knowingly.
- The replica is the real cost. A small Cloud SQL replica runs whether or not a failover happens. That is the honest price of the design, and it is the line item to check before you commit — for a solo builder, it is usually the largest cost in the whole setup.
- The load balancer and Cloud Armor policy are shared, not duplicated. One Cloud Armor policy with the same WAF and rate-limit rules protects both regions, so a failover does not quietly change your security posture.
The part most people skip is the failover test. An untested failover is a hope, not a plan. Once a quarter I run a controlled test: I mark the active region's health check unhealthy on purpose, watch traffic move, confirm the degraded flag and evidence_as_of appear in responses, then restore and watch traffic move back. The test takes about twenty minutes and has caught two real problems — a health check that depended on a cache instead of the database, and a firewall rule that the passive region's service account did not have. Both would have turned a real failover into a second outage.
What this does not protect against
Three honest limitations, because a failover design that hides them is a liability:
- It does not protect against a bad deploy. The same revision runs in both regions, so a bug ships everywhere at once. That is why the deploy pipeline matters more than the failover design — the pipeline is what keeps a bad revision from reaching either region.
- It does not protect against a data-plane bug in the application itself. If the scoring logic is wrong, both regions compute the same wrong score. Deterministic scoring helps here, because the same evidence always produces the same score, and a wrong score is at least reproducible and debuggable.
- It does not protect against a global control-plane incident. If the thing that is broken is Google Cloud's global load balancing itself, no region-level design saves you. That is a risk I accept, and I say so, rather than pretending two regions equal invincibility.
The takeaway
Multi-region failover for a small service is not a big-team project. It is a load balancer with a failover backend, a health check that actually checks the database, a cross-region replica with a named RPO, and a quarterly test that proves it works. The extra cost is dominated by one small database replica, and the extra code is a health endpoint, a degraded-mode flag, and a write queue.
The question to ask before building any of this is not "can I afford two regions?" — it is "what does my service promise when one region is gone?" If the honest answer is "a slightly stale score, clearly labelled, with writes queued," then this design is enough. If the answer has to be "the exact same data, the exact same writes, zero loss," then you need synchronous replication and a budget to match, and you should say that out loud before you start building. Failover is a trade-off you choose deliberately, not a checkbox you tick. Choose it on purpose, test it on purpose, and label what it serves.
