Read more of this story at Slashdot.
Backend
Microsoft Blames Monster Email Outage On Software Defect Introduced In July Fix
Slashdot0 views
Microsoft says last week's massive Microsoft 365 outage was caused by a software defect introduced in a July update that remained dormant until a Teams failover and a simultaneous 14% reduction in gateway capacity triggered a cascading failure. More than 80% of the affected gateway servers became unresponsive at the outage's worst point, and because the months-old change could not simply be rolled back, engineers spent roughly 15 hours developing and deploying a patch that restored the previous code path. The Register reports: According to a preliminary post incident report published by Microsoft on November 15, the defect was introduced in July 2026. The update looked fine at the time, Microsoft says, having passed load testing and been deployed without any impact. However, the testing wasn't good enough. Microsoft's report states: "Our load testing tended to focus on weaker hardware with fewer cores." The defect caused "lock contention inside an internal Windows process which caused the machine's resources (CPU & memory) to be exhausted." "The contention issue is worsened on machines with more cores and newer hardware." The fix containing the defect was prepared specifically for Microsoft 365 network gateway servers. Then two things happened at once. Teams failed over to the "primary" gateway that supports Exchange Online, creating a surge in load that was not, in itself, particularly unusual. At the same time, almost 14 percent of server capacity for the networking gateway was out of service, which wasn't a problem by itself either: it was on its way to being restored and, according to Microsoft, "the service had sufficient capacity headroom." Together, they pushed the gateway into the defect and chaos ensued. It was a classic cascading failure scenario. When a server loses connectivity, traffic gets routed to the next healthy endpoint. Which encountered the same problem. So traffic was rerouted again and... you can see where this was going. Within two hours, the defect was encountered across Microsoft's fleet. "At its lowest," the report states, "more than 80 percent of servers were unresponsive." So, what to do? Bringing servers back to service didn't help -- they were immediately overwhelmed and fell over again. "Our normally automated recovery processes were not designed to restore service from this broad unresponsive state." A straightforward rollback was also not possible because the defect was months old. Instead, engineers developed patch for the patch or, in Microsoft's words, "an update that returned the code path to the previously known good state in Windows." The patch was applied to groups of servers and service was gradually restored.
Source:https://tech.slashdot.org/story/26/11/16/1628256/microsoft-blames-monster-email-outage-on-software-defect-introduced-in-july-fix?utm_source=rss1.0mainlinkanon&utm_medium=feedPreviousUS Closes Loophole That Allowed China To Remotely Access Nvidia ChipsNextAnthropic cuts its coordinated vulnerability disclosure window from 90 to 30 days for high- and critical-severity issues, in response to AI-driven bug discovery (Anthropic)
Related posts
On Leaving Swift for Rust
0Lobsters
How Rust Changed Me As A Developer
0Lobsters

Dax Raad rebuilt his browser tool in Rust for $60. Then memory use fell 97%.
0The New Stack
Relationships Are Becoming Web3's Most Valuable Infrastructure
0HackerNoon
How to get emails for a Google Maps lead list without opening 500 websites
0dev.to
What Does an Arbitration in India Actually Cost?
0dev.to