Microsoft blames router IP address change for global outage The Register

[ad_1]

The worldwide outage of Microsoft 365 services that prevented some users from accessing resources for more than half a day last week was due to a packet bottleneck caused by a change in the router’s IP address.

Microsoft’s WAN knocked down a bunch of services beginning at 07:05 UTC on January 25 and although some regions and services were back online by 09:00, intermittent packet loss issues were not fully alleviated until 12:42 p.m. The wobble also affected Azure government cloud services.

In a post-mortem, Microsoft said changes to its WAN affected connectivity between customers and Azure, between regions, and between premises via ExpressRoute.

“As part of a planned change to update the IP address on a WAN router, a command given to the router caused it to send messages to all other routers in the WAN, which caused them all to recalculate their adjacency and forwarding tables During this recalculation process, routers were unable to properly forward packets passing through them.

“The command that caused the problem behaves differently on different network devices, and the command has not been verified using our full qualification process on the router it was run on.”

This meant that users were unable to access resources hosted in Azure or other Microsoft 365 and Power Platform services.

Microsoft said the monitoring systems detected DNS and WAN issues at 7:12 a.m., about seven minutes after they began.

By 8:20 a.m., Microsoft resident technicians had spotted the “problematic command that triggered the issues” and about 40 minutes later, network telemetry indicated that many services were working again.

However, Microsoft said the initial problem with the WAN meant that automated systems to maintain its health were suspended. This included systems to identify and evict faulty devices, as well as the traffic engineering system to optimize data flow on the network.

“Due to the pause of these systems, some network paths experienced increased packet loss beginning at 09:35 UTC until these systems were manually restarted, restoring the WAN to optimal operating conditions. This recovery has was completed at 12:43 UTC,” the post mortem added.

Microsoft’s efforts to make similar incidents less likely or severe include blocking “executing high-impact commands on devices” and requiring all command executions on devices to follow security guidelines.

The final post-incident report should be published fifteen days after the shutdown.

[ad_2]