Cloud Outage Headlines Blame the Region. The Checklist Starts at Quotas
Archi4 min read
Contents
- The Region Gets the Headline. The Quotas Get the Blame
- Quotas are per account, per region, per service
- DNS TTL is a failover timer you set months ago
- IAM drift builds without a single bad decision
- A snapshot is not a backup until you restore one
- The standby region is a different product catalog
- Egress addresses sit on someone else's firewall
- The bill headline is usually wrong
- Commitments expire on a calendar nobody watches
- Support tier sets the clock on everything else
- The runbook list for this week
The Region Gets the Headline. The Quotas Get the Blame#
When a cloud region degrades, the provider's postmortem reads like a weather report: power, cooling, a control plane loop. The customers who lost a day rarely failed at the same layer. Their failover plan assumed the second region behaved like the first, and it did not.
Below is a checklist for the parts that never make the provider blog. Each item carries a number, because the numbers decide whether the week stays boring.
Quotas are per account, per region, per service#
A region you barely use starts near zero. Default vCPU ceilings might permit 32 instances while production runs 400. Autoscaling in normal traffic never touches the ceiling, so nobody notices, until failover and every launch call returns a quota error. Then you learn that some increases need a business day and a support conversation.
Check: every quota production uses at 40 percent or more of its limit. File the increases in the standby region now. Write the approval lead time next to each one.
DNS TTL is a failover timer you set months ago#
A record with a 3600 second TTL lets resolvers hold a dead address for an hour. Health checks help only when the check interval times the failure threshold is shorter than your tolerance. Typical defaults are a 30 second interval and three failures, so 90 seconds before the record flips, plus whatever the client cached.
Check: the TTL on the record users actually resolve. Then confirm the mobile clients in the field respect it. Many do not.
IAM drift builds without a single bad decision#
A key from 2022 that still works. A role with a wildcard action added during a launch. A proof of concept service account that now writes to the production bucket. None of these appear in an outage report, and all of them slow a recovery, because nobody wants to run a script with credentials they cannot explain.
Check: keys older than 90 days, roles with wildcards, and any identity that has not authenticated in 60 days but still holds permissions.
A snapshot is not a backup until you restore one#
Restore time depends on the incremental chain, encryption key availability, and volume count. A quarterly restore into a scratch account produces a real number. Without it, your recovery time objective is a hopeful guess taped to a wiki page.
Check: last restore date, measured duration, and who ran it.
The standby region is a different product catalog#
Instance families arrive later in some regions. Managed services have gaps. A queue service may exist in one region and not the other, which turns a failover into an architecture change at the worst possible hour.
Check: every managed service in the dependency list against the standby region, monthly.
Egress addresses sit on someone else's firewall#
Failover changes your source IP. Partners who allowlisted your NAT addresses will drop traffic until their change ticket clears, which may be Monday. Collect those allowlists before you need them, and ask whether the standby region already has addresses registered.
Check: every partner that filters by IP, and the ticket turnaround each one quotes.
The bill headline is usually wrong#
Coverage of cloud costs centers on compute pricing. Finance gets surprised by egress, cross zone traffic, NAT gateway hours and per gigabyte processing, idle load balancers, and log storage growing at 4 percent a week. Data transfer direction should shape the architecture long before the invoice arrives.
Check: top five line items by growth rate, not by size.
Commitments expire on a calendar nobody watches#
Reserved capacity and savings plans lapse on their own schedule. When they do, the same workloads bill at on demand rates and the monthly total jumps 30 to 40 percent with no change in usage.
Check: renewal dates in the shared calendar, 60 days ahead.
Support tier sets the clock on everything else#
The plan you bought determines whether a severity one gets a human in 15 minutes or an email in four hours. During a regional event the queue is longer, and the provider triages by tier.
Check: contract tier, named contacts, and whether the on call engineer can open a severity one at 3 a.m. without an approval chain.
The runbook list for this week#
, File quota increases in the standby region for vCPU, elastic IPs, and load balancers., Lower the TTL on the user facing record to 60 seconds and watch resolver behavior for a week., Print every access key older than 90 days and delete the ones nobody claims., Restore one database and one volume into a scratch account, and time both., Compare the managed service list in each region, line by line., Register the standby region's egress addresses with partners that filter by IP., Put renewal dates and support tier contacts on the same page as the escalation tree.
The provider will publish its timeline within a week. Yours gets written by whoever was on call, and it will be shorter if the quotas were filed in March.
More articles