The 1.8-Second DNS Lag That Caused a 30% Drop in API Traffic

Archi4 min read

dnscloudincidentnetworking

Contents
  1. The 1.8-Second DNS Lag That Caused a 30% Drop in API Traffic
  2. The first hour: chasing the wrong ghost
  3. The record with the wrong TTL
  4. The fix was a single digit
  5. What I would test next time
  6. The numbers that matter
  7. What to check in your own stack

The 1.8-Second DNS Lag That Caused a 30% Drop in API Traffic#

On a Tuesday morning, the pager went off: API error rates had climbed from 0.2% to 4%. Not enough to trigger the emergency bridge, but enough to wake people up. I was on call. The first thing I did was pull up the latency dashboard. p99 had jumped from 180 ms to 1.8 seconds. That number looked familiar. It was almost exactly the default DNS timeout in our HTTP client.

The first hour: chasing the wrong ghost#

My first guess was a slow upstream dependency. The API calls a payment processor and a user profile service. I checked the payment processor's status page: all green. I checked our own database: query times were flat. I checked the load balancer: no spikes in 5xx. The only anomaly was that some requests were taking a full 1.8 seconds longer than others, and those requests were failing with a 504 from the upstream.

I pulled a packet capture from one of the API nodes. The pattern was clear: for about 30% of outbound requests to the payment processor, the TCP handshake never started. The node was sending DNS queries and getting no response. Then, 1.8 seconds later, it retried and got an answer. By then the request had already timed out.

The record with the wrong TTL#

We use a managed DNS service. I logged in and looked at the A record for the payment processor's API endpoint. It had a TTL of 30 seconds. That seemed normal. But when I queried the record from multiple resolvers, I got different answers. Some resolvers returned an IP in our region. Others returned an IP in a different region, one that had been decommissioned two weeks earlier.

The record had been updated during a failover test. The engineer changed the IP but left the TTL at 30 seconds. That should have propagated in half a minute. But our DNS provider's control plane had a bug: when you edited a record, the new TTL was applied to the authoritative zone, but the old TTL was still served by one of the anycast edges for up to 48 hours. The edge in our region was serving the old IP with the old TTL.

So for two days, roughly 30% of our API nodes were resolving the payment processor to a dead IP. The HTTP client's DNS timeout was 1.8 seconds. That matched the p99 latency exactly.

The fix was a single digit#

We changed the TTL on the record from 30 to 300 seconds. That forced the provider's control plane to push a full zone update to all edges. Within 10 minutes, every resolver was returning the correct IP. Error rates dropped back to 0.2%.

The root cause was not the DNS provider's bug. It was our own process: we had changed an IP without bumping the TTL first. A higher TTL would have made the stale record expire faster because the provider's edge would have flushed it sooner. Counterintuitive, but that is how their cache invalidation worked.

What I would test next time#

I ran a small experiment on a staging cluster. I changed a record's IP and left the TTL at 30 seconds. Then I queried from 20 different resolvers every 10 seconds. The stale IP persisted on 3 of them for 6 hours. When I bumped the TTL to 300 seconds before the change, the stale IP disappeared from all resolvers in under 5 minutes.

The lesson is not about DNS. It is about the smallest detail: the TTL is not just a cache duration. It is a signal to the provider's control plane about how aggressively to invalidate. A low TTL can actually slow down propagation if the provider's edge caches the old record longer than the TTL suggests.

We now have a runbook step: before any DNS change, raise the TTL to 300 seconds, wait for propagation, make the change, then lower it back. It adds 10 minutes to the process. It would have saved two days of degraded traffic.

The numbers that matter#

During the incident, we served 12 million API requests. About 3.6 million of them hit the stale IP and timed out. That is 30% of traffic. The average cost per failed request, based on downstream retries and support tickets, was roughly $0.004. Total cost: about $14,400. Not catastrophic, but enough to justify a 10-minute runbook change.

The p99 latency number, 1.8 seconds, was the clue that cracked it. It was not a random timeout. It was the exact DNS timeout configured in our HTTP client. That detail, a single configuration value, pointed directly to the resolver layer. Without it, I might have spent hours in the application logs.

What to check in your own stack#

If you run services that depend on external APIs, check three things. First, what is your HTTP client's DNS timeout? Second, what is the TTL on the records you depend on? Third, does your DNS provider have a documented invalidation behavior for record edits? Most do not, and that is the gap that bit us.

I also added a synthetic check: every 30 seconds, a probe resolves the payment processor's hostname from each region and compares the IP to the expected value. If they differ for more than two consecutive checks, we get an alert. That check would have caught this in 60 seconds.

The incident report is two pages long. The fix was one digit. The lesson is that in cloud infrastructure, the smallest details are often the ones that decide whether your traffic flows or stalls.

More articles