The Biggest Cloud Outages of 2025–2026, and What Their Post-Mortems Have in Common

When a major cloud provider goes down, a large slice of the internet goes with it: banking apps, games, airline check-in, smart doorbells, and thousands of websites that never knew which provider was underneath them. Speculation fills the first few hours. The useful information arrives days later, in the provider's own post-incident report.

This article covers four of the most disruptive outages of the past eighteen months, each with a detailed public post-mortem. It deliberately leaves out incidents where the provider hasn't published a root cause yet — guessing at those helps nobody.

The striking thing about all four is how similar they are underneath. None was a cyberattack. Each began with a routine internal change, spread far faster than anyone could react, and took hours to unwind.

What broke: Service Control, the system that checks API requests against policies and quotas across Google Cloud. When it fails, requests to dozens of Google Cloud products fail with it.

Root cause: A new quota-checking feature was added to Service Control on 29 May. In Google's words, the change "did not have appropriate error handling nor was it feature flag protected." It sat unused for two weeks. On 12 June a policy change containing blank fields was written to a Spanner table that replicates globally within seconds. Every Service Control instance read the blank fields, hit a null pointer, and crash-looped — in every region at once.

Why recovery was slow: Engineers found the cause in about ten minutes and had a kill switch rolled out in about forty. But when Service Control restarted in the large us-central1 region, every task hit the same Spanner table at the same moment. With no randomised backoff on retries, this "herd effect" overloaded the database, and that region took up to about two hours forty minutes to recover.

Google's fixes include feature-flagging critical code paths, making Service Control fail open, propagating replicated data incrementally rather than globally, and adding randomised exponential backoff.

What broke: DNS for dynamodb.us-east-1.amazonaws.com, the endpoint for Amazon's DynamoDB database in its largest region. DynamoDB sits underneath a great many other AWS services, so when its name stopped resolving, failures cascaded.

Root cause: DynamoDB manages its DNS records with two automated components: a DNS Planner that produces versioned plans, and a DNS Enactor, running in three Availability Zones, that applies them to Route 53. One Enactor was badly delayed. While it waited, another Enactor applied a much newer plan and then cleaned up old ones. The delayed Enactor finally applied its stale plan over the new one, and the cleanup then deleted that plan — leaving the regional endpoint with no IP addresses at all. In AWS's words, "all IP addresses for the regional endpoint were immediately removed."

Timeline (Pacific time): DNS failures began at 11:48 pm on 19 October. Engineers identified the DNS state as the cause at 12:38 am and restored it at 2:25 am — about two and a half hours.

Why recovery took until the afternoon: EC2's internal lease-management system depends on DynamoDB. By the time DynamoDB came back, so many leases had expired that re-establishing them all at once drove it into what AWS called "congestive collapse." Then network state changes backed up, and Network Load Balancer health checks began failing on new instances, removing healthy capacity. EC2 was fully recovered at 1:50 pm, more than fourteen hours after the first DNS failure.

AWS's fixes include disabling the DNS automation worldwide until the race condition is fixed, and adding a velocity limit on how much capacity a load balancer can remove when health checks fail.

What broke: Cloudflare's core proxy, which handles traffic for millions of websites. Visitors saw HTTP 5xx error pages across a large part of the web for about three hours. Cloudflare called it its worst outage since 2019.

Root cause: A permissions change on a ClickHouse database cluster, deployed at 11:05 UTC, made additional table metadata visible. A query that builds the Bot Management "feature file" didn't filter by database name, so it began returning duplicate rows and the file doubled in size. The proxy pre-allocates room for at most 200 features — about 60 were in use — and the oversized file blew past that limit. The Rust code handled the error with unwrap(), which panicked and stopped request processing.

Why it looked like an attack: The file was regenerated every five minutes, and only some database nodes had the permission change. So good and bad files alternated, and the network recovered and failed again in waves. Cloudflare's status page — hosted elsewhere — happened to go down at the same time. Engineers initially suspected a massive DDoS attack. It wasn't one.

Timeline (UTC): Impact on traffic began at 11:28. Generation of new feature files was stopped at 14:24, a known-good file was deployed by 14:30, and all services were restored at 17:06.

Cloudflare's fixes include treating its own generated configuration files with the same caution as user input, adding more global kill switches, and reviewing failure modes across the proxy.

What broke: Internet routes for customers using Bring Your Own IP (BYOIP) — organisations that announce their own IP address ranges through Cloudflare. When those routes were withdrawn via BGP, their addresses simply stopped being reachable.

Root cause: Cloudflare had automated the removal of BYOIP prefixes, a task previously done by hand, as part of a regularly running cleanup job. The job queried an internal API with a pending_delete parameter but no value. The API treated the empty value as "all BYOIP prefixes", and they were all queued for deletion. About 1,100 of 4,306 BYOIP prefixes — roughly a quarter — were withdrawn before an engineer stopped the job.

Timeline (UTC): Customer impact began at 17:56 and the job was terminated at 18:46. Customers could re-advertise their prefixes from the dashboard, and about 800 were restored that way by around 20:20. Roughly 300 more had lost their configuration on the edge servers and had to be restored by engineers, finishing at 23:03 — about five hours in total.

Cloudflare's fixes include stricter API schemas, separating configured state from operational state, and a circuit breaker that stops large, fast route withdrawals.

Configuration is code. Not one of these began with a code deploy in the traditional sense. A policy row, a database permission, a DNS plan, a cleanup job's API call — each was treated as data rather than software, and each bypassed the gradual rollouts and feature flags that code changes go through.

Global propagation is a double-edged sword. The same systems that let providers push a change worldwide in seconds also push a bad change worldwide in seconds. Three of the four post-mortems commit to slowing propagation down or adding brakes.

Empty values are dangerous. Blank policy fields at Google, an empty DNS record at AWS, an empty API parameter at Cloudflare. In each case "nothing" was interpreted as something drastic instead of being rejected.

Recovery is harder than the fix. Finding the cause took minutes to an hour. Getting back to normal took hours, because restarting systems all at once overloads whatever they depend on — the herd effect at Google, congestive collapse at AWS. Randomised backoff and throttled restarts appear in almost every remediation list.

DNS and BGP are the internet's single points of failure. Two of the four were, at the moment users noticed, simply names or routes that stopped existing. When that happens, everything built on top looks broken at once.

You can't prevent a hyperscaler outage, but you can reduce how badly it hits you and how fast you understand it.

• Know your dependencies. List the providers under every critical path: DNS host, CDN, cloud region, email provider, payment processor, authentication service. Many sites discover during an outage that their DNS, CDN and status page all live with the same company.
• Use sensible DNS TTLs. Very long TTLs slow down any failover you plan to do; very short ones increase load and the impact of a DNS provider blip. A few minutes to an hour suits most records. The DNS propagation guide explains the trade-off.
• Host your status page somewhere else. If your status page runs on the same provider as your site, it goes down exactly when people need it.
• Monitor from outside. Checks that run inside your cloud account can go down with it. An external monitor tells you what your users actually see.
• Retry with jitter. If your own code retries failed calls, add randomised exponential backoff, so your clients don't become the herd when a dependency comes back.
• Know how to diagnose quickly. Our guide to telling whether a site is down for everyone or just you walks through the checks in order.