Four Minutes of Bad DNS Cost Us 90 Minutes: A Cloud Postmortem
Archi4 min read
Contents
The Number That Mattered Was Four, Not Ninety#
When a cloud region lost a chunk of its traffic last month, the incident clock read 90 minutes. That is the number leadership saw. It is not the number that mattered. The number that mattered was four: the minutes between when a DNS record changed at the primary zone and when the secondary zone accepted it, leaving a window where resolvers received two different answers.
We had tested the change window. We had tested the failover path. We had not tested what happens when both run at once. This is a postmortem of a self-inflicted outage, and the lesson is narrower and more uncomfortable than "add more monitoring."
How the Morning Went#
The trigger was ordinary: an internal service needed a new subdomain for a staging cluster, and the platform team pushed the record through the standard pipeline. The pipeline validated syntax, applied the change, and reported success. At 09:41 local time, the first alert fired on elevated 5xx rates from the edge tier. By 09:43, two of the three edge points of presence were sending traffic to a backend that answered, but answered wrong.
What made this unusual was that the change succeeded. The zone file was valid. The API returned 200. The automated canary, which checks a single health endpoint every 60 seconds, stayed green through the whole four-minute window because the canary itself resolves through a cached stub and did not see the split answer.
Four Minutes Is a Long Time at Scale#
A four-minute split-horizon window sounds trivial. At our request volume it was not. In that window, roughly 2.1 million lookups hit resolvers that had the old answer, while a comparable volume hit resolvers with the new one. Because the new record pointed at a cluster that had not finished its own certificate rotation, clients that got the new answer saw TLS handshakes fail.
The failure was not the DNS change. The failure was that two independent operations, a record push and a certificate rotation, were scheduled into the same maintenance slot without a dependency check. Each was safe alone. Together they produced a four-minute window where the only correct answer was "wait."
Why the Detection Took 90 Minutes#
The edge alert fired in two minutes. The mitigation took 90 because nobody could agree on which system was lying. , The DNS console showed the new record as propagated., The certificate dashboard showed the rotation as complete., The edge logs showed a mix of 502 and 503 responses with no single cause., The runbook assumed DNS propagation was the slow part, so the first hour went into chasing a stale cache that did not exist.
The actual fix was to roll the record back to the previous value and let the rotation finish on its own schedule. That took one operator about four minutes once the team stopped assuming the problem was somewhere else.
What We Changed, and What It Cost#
We did three things, and none of them were new tooling.
A dependency calendar#
Record changes and certificate rotations now share one calendar view, and the pipeline refuses to apply a change tagged as network if a certificate job for the same target is running or due within 30 minutes. The check is a single API call against a shared state store. It has blocked four pushes in the past month, all of them legitimate collisions.
A canary that resolves fresh#
The health check now resolves through a fresh stub with cache bypass enabled, at 15-second intervals, against two different resolver paths. The cost is negligible: a few thousand extra lookups a day. The benefit is that the four-minute window would have been visible in roughly 30 seconds.
A named owner for "which system is lying"#
The 90-minute number came from debate, not from work. The runbook now assigns a single incident commander whose first job is to decide which subsystem is treated as authoritative, then stick to that call for 15 minutes before revisiting. We tested this in a tabletop exercise and cut simulated time to mitigation from 70 minutes to 22.
The Broader Point#
Cloud postmortems tend to focus on the largest number in the report. The 90 minutes is what gets attention in a review. The four minutes is what actually explains the incident, and it is the kind of number that rarely makes it into a dashboard because nothing was broken long enough to trigger a threshold.
Most of the headcount in our incident channel was spent on the assumption that DNS was slow. That assumption was reasonable in 2015. It is mostly wrong now, and it is wrong in a way that costs real time when an incident is live. Propagation gaps of a few minutes are common in multi-zone setups, and they are exactly the kind of thing that a health check with a 60-second interval and a cached stub will never see.
We keep a short list of numbers from this incident: four minutes of split answers, 2.1 million lookups in the window, 30 seconds of detection we now get instead of two minutes, and 22 minutes of simulated time to mitigation. None of those numbers are dramatic. The 90-minute figure will be the one someone remembers in a quarterly review. The four-minute figure is the one that should have been on the change ticket from the start.
More articles