On Nov. 15, 2026, AWS reported problems affecting several services in its US-East-1 region. According to the AWS Health Dashboard, customers saw authentication, account registration and connectivity errors, which one tracker puts at 3 hours 16 minutes. AWS said the affected services recovered but did not establish a definitive root cause in its incident update. You can read the tracker timelines at TechRepublic and Control Plane.
This article uses that event as a working example. It covers what a NOC can do before, during and after a regional failure, whatever the provider and whatever the cause turns out to be.
How do we tell a provider outage from a problem inside our own or a client's network?
Compare what fails with what still works. If many unrelated clients, sites and ISPs show the same failure against the same provider endpoints while other internet destinations respond normally, suspect the provider. If one site or one circuit is affected, suspect the local network.
A short decision routine helps:
- Check the provider's status page and health dashboard, but do not wait for it. Status pages often lag what your probes see.
- Test the same target from several vantage points: a client site, your own NOC, and a probe on a different carrier.
- Test a neutral destination from the same place. If it works, the local path and DNS are probably fine.
- Look at the error type. Timeouts at the network layer point to routing or links. Fast, well-formed errors from the service (authentication denied, service unavailable) point to the provider.
- Check recent changes on your side: firewall rules, certificate renewals, DNS edits, SD-WAN policy pushes.
The pattern of failure is the strongest signal. A provider problem looks the same across clients who share nothing but the provider. This is the core of separating provider outage from network issue.
Which synthetic checks and external probes catch a cloud sign-in failure early?
The most useful checks exercise the full sign-in path, not just whether a port answers. A scripted login to a test account, run every minute or two from outside your network, will often flag an authentication failure before users start calling the helpdesk.
Build a layered set:
- Authentication checks: a scripted sign-in with a dedicated test identity against the console, the identity provider and any single sign-on flow your clients rely on. Record time to complete as well as pass or fail.
- API checks: a lightweight authenticated call to the key services each client uses, in each region they depend on.
- Registration and token checks: where clients create accounts or request tokens, test that step too, since it can fail while existing sessions keep working.
- DNS and TLS checks: confirm names resolve and certificates validate, so you can rule out the easy causes quickly.
- External probes: run the same checks from at least two networks outside any client environment, ideally in different geographies.
Keep test credentials least-privilege and separate from real users. Alert on consecutive failures from multiple probes rather than a single miss, to cut noise. For anyone asking how to monitor an AWS outage, the answer is mostly this: watch the user path from outside, and treat the provider's dashboard as confirmation, not the trigger.
What should a NOC do in the first 15 minutes of a cloud-region incident?
Confirm the scope, open one parent incident, and start client communication. In the first 15 minutes the goal is not to fix the provider. It is to know who is affected, rule out your own changes, and give clients an honest first update.
A practical sequence:
- Minutes 0 to 3: acknowledge the first alerts and check whether the failing probes are in the same provider and region.
- Minutes 3 to 6: run the vantage-point tests above and check the provider health dashboard and your change log.
- Minutes 6 to 9: declare a parent incident, link every related client alert to it, and assign one owner and one communicator.
- Minutes 9 to 12: list affected clients and workloads using your asset and dependency records. Flag any that are life-safety, revenue-critical or regulated.
- Minutes 12 to 15: send the first client notice, update the internal status channel, and set the next update time.
Avoid making changes in client environments just to be seen doing something. Failing over, rotating keys or rebuilding sign-in settings in the middle of a provider fault can create a second incident. Only act on a documented runbook step.
What do we tell clients while the provider has no root cause yet?
Tell them what you know, what you do not know, what you are doing, and when you will update next. Do not guess at a cause. In the Nov. 15 event, AWS reported recovery without a definitive root cause in its incident update, so any firmer claim would have been speculation.
Good MSP outage client communication is short and consistent:
- Impact: which services or sign-ins are affected, in plain words.
- Source: that the evidence points to the provider, and where you saw it.
- Your actions: monitoring, workarounds that are safe, and escalation with the provider.
- Next update: a specific time, even if the message is that nothing has changed.
Avoid blame, jargon and promises about when service will return. After recovery, send a closing note that states the timeline you observed and says plainly if the provider's cause is still unconfirmed. Offer a review of any client dependencies that made the impact worse.
How do we reduce dependence on a single region or identity path?
Start by mapping what each client actually depends on, then remove the single points of failure that matter most. Full multi-region design is costly, so prioritize the workloads where a few hours of sign-in failure would hurt the business.
Options, from lightest to heaviest:
- Break-glass access: keep emergency administrator accounts that do not depend on the same identity path, stored securely and tested on a schedule.
- Cached or offline-tolerant sign-in: where the application allows it, so existing sessions survive a short authentication failure.
- Secondary identity path: a documented alternative for admins to reach critical systems if the primary provider region is degraded.
- Regional redundancy for critical workloads: run or replicate them in a second region, and test the failover rather than assuming it works.
- Dependency records: know which control-plane features, such as account creation or token issuance, sit in one region even when workloads run elsewhere.
Write these into a cloud outage response plan, assign owners and rehearse it. A plan nobody has exercised tends to fail on the first real day.
How should alerts be triaged when many clients are hit by the same upstream failure?
Group them under one parent incident and triage by client impact, not alert volume. When one upstream failure fires hundreds of alerts, the risk is that the NOC drowns in duplicates and misses an unrelated fault.
What works in practice:
- Correlate: tie alerts to the shared provider and region, and suppress or collapse child alerts under the parent.
- Rank by criticality: use each client's agreed priority levels, so clients with critical workloads get attention first.
- Keep a separate lane: assign one engineer to watch for non-related alerts so a genuine local failure is not hidden by the noise.
- Use templates: pre-written updates per client tier save time and keep messaging consistent.
- Track recovery per client: the provider may report recovery before every tenant is working again, so verify with your own checks before closing.
For MSPs, white-label support can help absorb the surge of tickets and calls during a wide event. Good 24/7 NOC cloud monitoring also depends on having the contract-defined response targets for each priority level clear before the incident starts.
After the incident: what to review
Once service is stable, hold a short review. Check how long it took your probes to detect the failure compared with the provider's first notice, whether alerts were correctly grouped, and whether client messages were timely. Update runbooks and dependency records with what you learned.
Frequently asked questions
Should we wait for the provider's status page before declaring an incident?
No. Declare an incident when your own external probes and multiple client reports agree. Status pages can lag, and early internal confirmation lets you start client communication sooner.
Can a monitoring tool hosted in the affected region fail with it?
Yes. If your monitoring, alerting or ticketing runs in the same region or identity path, it may be impaired exactly when you need it. Host probes and notification channels in separate regions or providers.
Is multi-region deployment always worth it?
Not for every workload. Weigh the cost and complexity against the business impact of an outage, and use multi-region design for critical systems while others rely on tested recovery steps.
How often should we test break-glass accounts?
On a regular schedule agreed with each client, and after any change to the identity setup. The point is to confirm the accounts work and that the right people know where the credentials are.
What if the provider never publishes a root cause?
Report what you observed: timing, scope and recovery. Say clearly that the provider has not confirmed a cause, and focus the follow-up on reducing the dependencies that increased impact for your clients.
This article was written with AI and reviewed by our team. It was first published on hex64.net.

