Introduction
Building a Landing Zone (LZ) at scale is a beast of a task that requires a deep understanding of your chosen cloud provider. The industry best practice? Use Infrastructure as Code (IaC) all the way.
Think of your cloud provider as a massive community pool. As a platform engineer, your job is to organize that pool: you throw in the floaties (policies), set up the swimming lanes (networking), manage the cantina (billing), and generally keep the chaos in check.
But what happens when one of your clients suddenly needs a new project? Or begs you to open a firewall rule right now?
As an experienced engineer, you obviously have automations for all of this. But if the client still has to ping you on Slack to trigger them, you’re the bottleneck. This is where self-service comes into play. In this article, I want to walk you through a backend architecture we built at my company to power a reliable, scalable self-service portal.
The Problem
In my company, GCP is our bread and butter. As platform engineers, we followed the FAST Fabric paradigm to build our LZ with Terraform. Because of this, we ended up with a multi-step, multi-state LZ.
Let’s make this clearer with a simplified example we’ll use throughout the article. Imagine our LZ code is structured like this:
- 0 — Bootstrap: Sets up the organization, billing, etc.
- 1 — Networking: Sets up the shared VPCs, hub-and-spoke architecture, and routing.
- 2 — CI/CD: Sets up a unified CI/CD solution (e.g., Cloud Build worker pools or self-hosted GKE runners).
- 3 — Project Factory: Creates the actual application projects.
Each of these steps might contain one or more states. Because we have a massive LZ, each of these states takes about 5 — and in extreme cases up to 15 — minutes to apply.
Now, imagine our self-service backend receives two different, independent requests:
- Create a VPC: This automation touches Step 1.
- Create CI/CD runners: This automation needs to create a VPC for the runners, so it touches both Step 1 and Step 2.
In a perfect world, we could just run these sequentially. The backend updates and applies the YAMLs for the VPC, then updates and applies the YAMLs for the runners. Done in maybe 20–30 minutes. Easy.
But as we grow, more and more clients start using our LZ. Awesome, right? Until the day we feared most arrives: our self-service backend suddenly gets hit with 100 different requests across all steps. Running all these sequentially in a queue would take an eternity, leaving our clients incredibly frustrated. In fact, we just might be able to create a whole new LZ with Claude Code by the time the 100th automation finishes applying.
We needed a parallelism mechanism. Classic.
The Prototype
Let’s take our two requests from before (VPC, CI/CD) and step back to a time before we had advanced automations. Easy — we’re senior engineers. We can just write a few YAMLs, toss them into the right factories, run our applies, and boom — everything is golden. Multi-step, multi-state deployments solved, right? Except, of course, this completely falls apart at scale.
Imagine we have an engineer on call 24/7 just to receive requests. To save time, our engineer collects all incoming requests within a 10-minute window. Every 10 minutes, they stop taking new requests, queue them up, run terraform apply for the batch, and once it finishes, they move on to the next batch.
Let’s look at the microservice High-Level Design (HLD) for this:
- Main Backend: Receives requests, handles the 10-minute apply lock, queues requests, and talks to the database.
- Automation Microservice: Adds the relevant YAMLs to the right step and pushes to Git.
- TF Applier: Runs the actual apply for each state while the lock is active.
Prototype
The Flow:
- 0: An incoming request hits the system.
- 1: The main backend checks if the lock is active.
- 2: If locked, it queues the incoming request and runs terraform apply on changes pushed to Git before the lock kicked in.
- 3: If unlocked, it calls the automation microservice for the queued requests and any new incoming ones.
Why This Sucks
Let’s be real — our backend here behaves way too much like a human taking coffee breaks. While we managed to batch our changes, we didn’t actually achieve parallelism.
There’s a ton of wasted time here. The real atomic operation is the apply itself. Every time we queue a request and wait for a 10-minute window, we lose valuable time. Think about it: absolutely nothing should stop us from preparing the next batch of YAMLs while an apply is currently running!
Initially, we thought we had to wait for the apply to finish so Git could remain our single source of truth. But saving states to a DB allows us to infer what the current state of the LZ is. We can do way better than this. Back to the drawing board!
Version 2: The Event-Driven Approach
We had an interesting observation: we can constantly prepare and push YAMLs. There will come a time when they get applied, but nothing should stop us from continuously mutating the desired state in Git.
What if our Terraform application process wasn’t a cron job, but one continuous, event-based chain of operations?
Here is the overall flow:
- Main Backend: Receives requests continuously. No more locks! It instantly calls the automation microservice.
- Automation Microservice: Updates the YAMLs and pushes to Git. But here is the crucial part: it also updates the DB with exactly which steps and states it touched (e.g., “I just updated Step 1 / State A, and Step 2 / State B”).
- TF Manager (New Microservice): Its sole purpose is to run one long, continuous chain of applies. It runs a loop: in each iteration, it checks the DB for the lowest pending serial (step and state) and calls the applier.
- TF Applier: Runs terraform apply and saves the logs to a bucket.
Let’s look at the updated Microservice HLD:
Version 2
Why this is a game-changer:
We essentially built a continuous, non-stopping line of state resolution.
Let’s revisit our classic example (creating a VPC and creating CI/CD runners). This time, let’s flip the script: the create-runners request comes in just before the create-vpc request.
- 1: The backend gets the create-runners request. The automation pushes YAMLs for Steps 1 and 2 to Git, logs it in the DB, and wakes up the TF Manager.
- 2: The TF Manager sees Step 1 is the lowest pending state and starts applying it.
- 3: While Step 1 is applying, the create-vpc request comes in. Its automation pushes a new NewVPC.yaml to Git for Step 1 and logs it in the DB.
- 4: The TF Manager finishes its apply. It looks at the DB to see what’s next.
- 5: Instead of moving on to Step 2, the TF Manager sees that Step 1 has pending changes again (thanks to the VPC request that just snuck in). It pulls the latest code and applies Step 1 again.
- 6: Once Step 1 is fully resolved and cleared from the DB, the TF Manager moves on to the lowest remaining state: Step 2 (for the runners).
- 7: All changes are applied, and the TF Manager goes back to sleep.
Instead of waiting for create-runners to be 100% finished before even starting the YAML generation for create-vpc, we created an intelligent, self-healing loop. We only advance to a higher step if there is nothing lower left to run. This aligns perfectly with the FAST Fabric paradigm!
Sure, create-runners took slightly longer to finish because it got preempted by lower-level state changes, but the overall throughput for both automations was vastly shorter. This enables us to handle a massive load of concurrent requests with a highly reasonable SLA.
Wrapping it up
In this article, we went over the High-Level Design of an apply mechanism we built for our self-service portal. I intentionally skipped over a few complex headaches — like execution history, IAM, soft/hard locks on Terraform states, and rollback mechanisms — to keep the core concept clear.
That being said, we believe this architecture is the perfect foundation for building a scalable, resilient, and observable self-service backend. And to prove that an elegant design doesn’t require a massive codebase: every single microservice we built for this clocked in at under 200 lines!
As cloud infrastructure matures, developer self-service is a crucial capability that every mid-to-large company will need to master. I’d love to hear about your experiences building similar architectures, or how you handle Terraform at scale, in the comments!
Finally, I feel obligated to shout out my fellow platform engineers for designing this architecture alongside me. I definitely couldn’t have done it without them. Thank you!
Originally published on Medium.
