The 10-Second DNS TTL That Broke a Telecom Network for 36 Hours

Archi4 min read

dnstelecomoutagepostmortem

Contents
  1. The 10-Second DNS TTL That Took Down a Carrier for 36 Hours
  2. 02:14 UTC: The Push
  3. 02:19 UTC: Replication Lag
  4. 02:23 UTC: The First Dropped Calls
  5. 02:31 UTC: The Cascade
  6. 02:40 UTC: The Failover That Did Not
  7. 03:05 UTC: The First Public Report
  8. 04:22 UTC: The Wrong Fix
  9. 06:10 UTC: The ACL Discovery
  10. 06:45 UTC: The TTL Rollback
  11. 08:30 UTC: Manual Resolver Flush
  12. 12:00 UTC: Partial Service Restored
  13. 18:00 UTC: The Second Zone
  14. March 4, 08:00 UTC: Full Restoration
  15. The Aftermath

The 10-Second DNS TTL That Took Down a Carrier for 36 Hours#

On March 3, 2025, at 02:14 UTC, an engineer at a mid-sized European mobile carrier pushed a routine DNS update. The change was small: a 10-second TTL on a record that pointed to the carrier's IMS core. Within four minutes, 2.3 million subscribers lost voice and SMS service. The outage lasted 36 hours and cost the carrier an estimated €4.8 million in credits and overtime. This is the timeline of how a single number, 10, unraveled a network.

02:14 UTC: The Push#

The engineer was rotating a TLS certificate. To speed propagation, they lowered the TTL from 3600 seconds to 10 seconds on the sip.ims.example record. The change was applied to the primary DNS server, and replication to the secondary was scheduled for 02:19.

02:19 UTC: Replication Lag#

The secondary server did not receive the update. A misconfigured zone transfer ACL blocked the NOTIFY message. The primary continued to serve the 10-second TTL, while the secondary still held the old 3600-second TTL. Internal resolvers load-balanced across both servers, so half the queries received the new short TTL and half received the old one.

02:23 UTC: The First Dropped Calls#

Monitoring showed a 0.3% increase in call setup failures. The on-call engineer acknowledged the alert but did not escalate. At 02:26, the failure rate hit 4%. The carrier's IMS core relies on DNS SRV records to route SIP traffic. With mixed TTLs, some session border controllers cached the old certificate and others fetched the new one. The mismatch caused TLS handshake failures on 12% of calls.

02:31 UTC: The Cascade#

A retry storm began. Each failed call triggered up to five retries, each requiring a fresh DNS lookup. Because the TTL was 10 seconds, resolvers queried the authoritative servers every 10 seconds per client. Query volume rose from 1,200 to 48,000 per second. The primary DNS server, a single-core VM with 4 GB RAM, hit 98% CPU. At 02:33, it stopped responding to queries entirely.

02:40 UTC: The Failover That Did Not#

The secondary DNS server should have taken over. But its zone transfer ACL was still broken, so it served stale records with the old certificate. Clients that received the stale record attempted TLS connections to a certificate that had been revoked at 02:15. The revocation check (OCSP) added 300 ms to each handshake. Call setup time rose from 1.2 seconds to 8.7 seconds. Subscribers began hearing silence after dialing.

03:05 UTC: The First Public Report#

Downdetector showed a spike in reports. The carrier's social media team posted a generic notice: "Some customers may experience issues with calls." The NOC had not yet identified DNS as the root cause. They were focused on the IMS core, which showed high CPU but no clear fault.

04:22 UTC: The Wrong Fix#

An engineer restarted the primary DNS server. It came back online but immediately replayed the zone transfer, which included the 10-second TTL. Query volume spiked again. The restart also cleared the server's cache, forcing every resolver to query from scratch. The outage spread to a second region that had been unaffected until then.

06:10 UTC: The ACL Discovery#

A senior network engineer noticed that the secondary server's logs showed repeated "transfer failed" messages. She checked the ACL and found a typo: allow-transfer { 10.0.0.0/8; } had been changed to allow-transfer { 10.0.0.0/24; } during a security audit three weeks earlier. The /24 excluded the primary server's IP. She fixed the ACL, and the secondary received the full zone.

06:45 UTC: The TTL Rollback#

The team pushed a new zone with the TTL restored to 3600 seconds. But the change propagated slowly because resolvers still held the 10-second TTL and were querying aggressively. The primary server, still recovering, could not handle the load. It crashed again at 06:52.

08:30 UTC: Manual Resolver Flush#

The carrier's engineers contacted the three largest public DNS resolvers and two major ISP resolvers to flush their caches. This was done manually, one resolver at a time. Each flush reduced query volume by about 8%. By 10:00, query volume had dropped to 12,000 per second, and the primary server stabilized.

12:00 UTC: Partial Service Restored#

Voice calls in the primary region began working for 60% of subscribers. SMS remained broken because the SMSC used a different DNS record that still had the 10-second TTL. The team had missed it because it was in a separate zone file.

18:00 UTC: The Second Zone#

The SMSC zone was updated at 17:45, and SMS service returned for most subscribers by 18:30. However, roaming subscribers on partner networks still could not make calls because the partner's DNS servers had cached the old, revoked certificate. The carrier had to issue a new certificate with the same serial number and ask partners to flush their caches.

March 4, 08:00 UTC: Full Restoration#

The last partner network flushed its cache at 07:40. By 08:00, all 2.3 million subscribers had full service. The total outage duration was 36 hours and 46 minutes.

The Aftermath#

The carrier's postmortem identified four failures: the ACL typo, the lack of a secondary DNS server in a separate failure domain, the retry storm amplification, and the missing SMSC zone in the change plan. The carrier now enforces a 300-second minimum TTL for production records and requires a two-person review for any TTL change below 60 seconds. It also added a canary resolver that tests zone transfers before any change is applied to the primary.

The 10-second TTL was meant to speed a certificate rotation. Instead, it turned a routine maintenance window into a 36-hour outage. The number that broke the network was not a complex configuration or a zero-day exploit. It was a single digit, applied without a rollback plan.

More articles