Microsoft reveals the root cause of large-scale Microsoft 365 outages: three factors superimposed and more than 80% of gateway servers lost response

📅 2026-11-16

Abstract:

Microsoft announced the preliminary investigation results of the widespread Microsoft 365 outage from November 12 to 13, saying that the failure was caused by a Windows server flaw that had been dormant for several months, Teams traffic switching, and a reduction in the available capacity of the gateway. The accident lasted from 16:20 UTC on November 12 to 16:10 UTC the next day, nearly 24 hours.

Microsoft said the flaw originated from a fix deployed for Microsoft 365 Network Gateway Server in July this year. Under a specific load, lock competition occurs in Windows internal processes, and the server's CPU and memory resources are gradually exhausted, and eventually stop responding. The problem had previously gone undetected, in part because Microsoft's test environment used hardware with fewer cores; the investigation found that lock contention was more likely to intensify on newer servers with more cores. Microsoft emphasized that this flaw and related fixes only involve Microsoft 365 gateway infrastructure and do not affect other Windows deployments.

Approximately 15 hours before the failure, Teams traffic switched from the backup network gateway to the primary gateway shared with Exchange Online, changing traffic patterns and increasing load; at the same time, approximately 14% of the gateway's server capacity was temporarily unavailable due to another issue that occurred the previous day. Microsoft said that these two factors alone did not have an impact, but when combined with latent flaws, they triggered server overload.

After the server loses response, the traffic management system transfers the request to other normal nodes, which puts the remaining servers under greater pressure and the fault spreads one by one. In less than two hours, more than 80% of the gateway servers failed to respond; the protective measures originally used to isolate faults in different regions failed to prevent the problem from spreading across regions. Microsoft said that the most serious incident during the incident was Exchange Online, where many users were unable to access their mailboxes or send and receive emails normally; services that relied on Exchange functions such as Teams, Copilot, SharePoint, and OneDrive were also affected to varying degrees, and chat and conferencing functions of some services could still be used.

The recovery process was drawn out because the restarted server could become unresponsive again as soon as it received the backlogged traffic, and because the flaw was deployed months ago, regular rollbacks and feature toggles could not be undone quickly. Engineers eventually restored the connection processing code to the previous stable version, then repaired it in batches, restarted the server, and gradually restored traffic. Microsoft stated that global webmail services were restored to expected levels at 14:17 on November 13, and it was confirmed at 16:10 that day that users were no longer directly affected by the incident; delayed emails were subsequently processed.

Microsoft has increased server capacity and monitoring, added traffic distribution protection measures, and extended mail delivery waiting times to prevent backlogged mail from being misjudged as delivery failures. The company said there is currently no indication that the incident involved malicious activities or artificial intelligence agents, and plans to issue a final post-event report before November 20 to supplement long-term corrective measures.

Related tags

Related articles

Comments

0/500
Captcha (click to refresh)
No comments yet