Part 3 of 3
The framework from Part 1 (three business decisions) and Part 2 (progressive validation) works at any pace. You can apply it over months as part of normal operations, or compress it into a structured engagement that moves from undefined targets to a validated recovery commitment in weeks. This post walks through how to run that engagement yourself, with concrete steps, outputs, and a before-and-after example from a real enterprise engagement.
Step 1: Kickoff (define the scope)
Bring three groups into one room: the application owner who knows the technical architecture, the business owner who understands the consequence of downtime, and the platform team that owns the shared services your application depends on. If a dependency is owned by a team not represented in the room, that’s your first action item before proceeding.
Ask these questions in order:
- What does the user need to be able to do for this function to count as “recovered”?
- Which applications, shared services, and accounts are in the path of that user journey?
- Who owns each piece, and have they confirmed their recovery timeline?
The output artifact is a boundary map: a document listing every service and account in the recovery path, with an owner assigned to each and their stated recovery timeline recorded. This single artifact becomes the foundation for every subsequent step.
Step 2: Assessment (surface the gaps)
Run a structured review of each component in the boundary map. For each dependency, ask: does it recover in the target AWS Region, does it recover within the stated recovery time objective (RTO), and does an upstream service’s recovery target cap the RTO you can actually commit to?
- A Well-Architected Framework Review against the Reliability pillar gives you a systematic way to surface single points of failure, missing health checks, and unverified quotas.
- An Operational Readiness Review checks whether the runbook’s assumptions hold: are the resources pre-provisioned, do upstream dependencies recover within the target, and are there any manual steps that depend on a specific person being available?
The output is a prioritized list of dependency gaps with owners assigned and a target fix date for each. “Done” at this step means every gap has an owner, a planned fix, and a committed date. Gaps without owners don’t get fixed.
Step 3: Validation plan (structure the tests)
Build a test calendar that progresses through the levels from Part 2. A practical sequence for a first engagement:
- Week 2: Tabletop exercise on the highest-consequence function. Confirm ownership, escalation path, and expected duration per step.
- Week 4-6: Planned failover after gap remediation. Measure actual recovery time against the committed RTO. Verify data consistency against the recovery point objective (RPO).
- Week 8-10: Surprise drill (if the planned failover passed). Test coordination without advance notice.
Don’t schedule a game day with fault injection until you’ve passed a planned failover cleanly. Each level builds confidence for the next. Attempting a higher level before the lower one passes cleanly wastes effort and produces noisy findings that obscure the real gaps.
Step 4: Execute and learn (close the loop)
Each test produces findings. Document them in a format that maps each finding back to the decision it informs:
- Boundary findings (a shared service discovered mid-test that wasn’t in the recovery scope) → feed back into Decision 1.
- Consequence findings (recovery took longer than the business can tolerate) → feed back into Decision 2, where the business owner either accepts the longer timeline or invests in a faster architecture.
- Dependency findings (an upstream service whose recovery target capped the application’s own, a resource that wasn’t pre-provisioned) → feed back into Decision 3.
The output is an updated recovery commitment that reflects what you actually measured, not what you assumed. This document replaces the prior version and becomes the baseline for the next cycle.
Step 5: Ongoing cadence (prevent drift)
Recovery posture drifts as applications change. A new microservice adds a dependency nobody mapped. A team deploys a configuration change that shifts traffic routing. A quota that covered peak load 6 months ago no longer does. Schedule a quarterly review that checks: has the boundary changed (new shared services, new accounts)? Has the business owner confirmed they still accept the RTO/RPO targets? Have any dependencies changed their recovery behavior?
A trigger for re-engagement outside the quarterly cycle: any architecture change that adds a cross-account dependency or changes the failover path.
What this looks like in practice
In the enterprise engagement referenced throughout this series, the team applied this pattern across its portfolio of applications and accounts. The first exercise missed a contractual recovery commitment of 5 hours, taking more than twice that long. Root causes mapped to all three decisions: the boundary excluded a shared authentication platform (most downstream applications couldn’t authenticate users), the dependency chain included an upstream service whose recovery target capped what the applications calling it could achieve, disaster recovery (DR) resources weren’t pre-provisioned, and DNS failover depended on a manual step that took over 3 hours under coordination pressure.
Over 4 months, the team redefined the boundary to include shared platforms, documented the full dependency chain per application, reconciled each application’s target against its slowest dependency, pre-provisioned the recovery environment, and replaced manual DNS failover with automated routing built on Amazon Application Recovery Controller (ARC) routing controls. ARC routing controls are on/off switches on a data plane that shifts traffic without depending on the Amazon Route 53 control plane, so a cutover doesn’t rely on modifying DNS records mid-event. This follows the Well-Architected guidance to rely on the data plane, not the control plane, during recovery (REL11-BP04), the same static-stability principle behind pre-provisioning. The second exercise passed. The organization operated a live customer-facing system from the recovery Region for the first time. Senior leadership cited the structured engagement as the factor that connected the two different outcomes.
Accelerating the timeline
You can run every step in this post with your own teams. If you want to compress the timeline, AWS Enterprise Support provides specific mechanisms that map to each step: a Technical Account Manager to facilitate the kickoff and maintain engagement momentum between sessions, Well-Architected and Operational Readiness Reviews for the assessment, specialist Solutions Architects for deep dependency analysis, and AWS Countdown for coordinated operational-readiness and engineering support during critical events, including planned failovers and DR game days. These are acceleration options, not prerequisites. The steps and outputs are the same either way.
The AWS Resilience Lifecycle Framework and AWS Resilience Analysis Framework provide the detailed practices behind each decision, and AWS Resilience Hub helps you assess applications against your defined RTO and RPO targets.
