The 200-Millisecond DNS Retry That Cost Cloud Customers $2.4M a Year
Archi3 min read
Contents
- The Number Nobody Budgeted For
- Backstory: A Default Born in 2016
- 2019: The Anycast Rollout
- A Topology Shift
- First Signals
- 2021: Containers Multiply the Queries
- The Query Explosion
- Retry Amplification
- 2022: The Bills Cross a Threshold
- The Audit That Started It
- Reconstructing the Cost
- 2023: The Fix That Took Four Months
- Month One: Change the Default
- Month Two: Find the Legacy Clients
- Month Three: Cache Tuning
- Month Four: The Rollout
- What the Timeline Shows
- Defaults Age Badly
- Costs Hide in Distributed Line Items
- Small Values Scale Linearly
- The Current State
The Number Nobody Budgeted For#
A single DNS resolver setting, a retry interval of 200 milliseconds, became one of the more expensive default values in cloud infrastructure. Over one fiscal year, a mid-sized managed services provider traced roughly $2.4 million in avoidable spend to it. The full timeline, reconstructed from engineering notes and billing exports, shows how a value chosen for resilience turned into a recurring tax.
Backstory: A Default Born in 2016#
When the resolver was first configured, the 200-millisecond retry was reasonable. Networks were slower, resolvers sat closer to clients, and a fast retry masked dropped UDP packets. Nobody revisited the number when the topology changed.
2019: The Anycast Rollout#
A Topology Shift#
The provider moved its resolvers to anycast addresses across six regions. Latency between a client and its nearest resolver dropped, but the number of in-flight retries per query rose. The retry value stayed at 200 milliseconds.
First Signals#
Engineers noticed a small uptick in resolver CPU. It was logged as noise. The monthly cloud bill rose by less than 1 percent, which was easy to attribute to growth.
2021: Containers Multiply the Queries#
The Query Explosion#
As workloads moved to containers, each pod spun up its own resolver stub. Query volume grew faster than request volume. A service making 400 requests per second could generate thousands of DNS lookups per second during connection churn.
Retry Amplification#
When a lookup missed the local cache, the resolver fired retries at 200-millisecond intervals. Under load, those retries stacked. One failed upstream reply could trigger a burst of four to six duplicate queries. The billing team saw egress and resolver compute creep upward, but no one tied it to DNS.
2022: The Bills Cross a Threshold#
The Audit That Started It#
A customer complained about a 34 percent year-over-year increase on a workload that had not changed. Finance pulled the invoice and found that DNS-related line items were scattered across compute, network egress, and a managed resolver service. Nobody owned the total.
Reconstructing the Cost#
Engineers replayed a week of production traffic through a staging resolver. They measured 1.9 billion queries in seven days, with 41 percent marked as retries. At the contract rate for resolver capacity and the egress cost of forwarded queries, that retry traffic alone came to about $200,000 per month across the customer base.
2023: The Fix That Took Four Months#
Month One: Change the Default#
The obvious move, raising the retry interval to 1 second, cut retry volume by 78 percent in staging. It also exposed a second problem: a handful of legacy clients depended on fast retries to reach a secondary name server that was misconfigured. Those clients started timing out.
Month Two: Find the Legacy Clients#
A scan of resolver logs identified 212 client subnets hitting the broken secondary. Most belonged to three internal applications and one partner integration. The team patched them one at a time, over three weeks.
Month Three: Cache Tuning#
The larger win came from cache behavior. Negative caching was effectively disabled, so failed lookups were retried immediately by every client. Enabling a 30-second negative cache TTL removed another 22 percent of resolver load.
Month Four: The Rollout#
The new defaults shipped behind a feature flag, region by region. The first region saw retry traffic drop from 41 percent to 6 percent within a day. By the end of the month, the annualized savings estimate landed at $2.4 million.
What the Timeline Shows#
Defaults Age Badly#
The retry value was not wrong when it was set. It became wrong when the network changed underneath it. Cloud defaults inherit the assumptions of the year they were written.
Costs Hide in Distributed Line Items#
DNS spend rarely appears on one invoice line. It shows up as resolver compute, forwarded query egress, and client CPU spent waiting. The audit only worked because someone added those lines together for the first time.
Small Values Scale Linearly#
A 200-millisecond retry multiplied by billions of queries per week is not a rounding error. The cost per query is tiny. The query count is not.
The Current State#
The provider now publishes resolver retry and cache settings in its internal service catalog, with a quarterly review. Two other teams have since found similar retry amplification in HTTP client libraries, where a 250-millisecond backoff default was driving duplicate requests to rate-limited APIs. The pattern repeats: a small resilience setting, chosen years ago, quietly compounding against a workload that no longer resembles the one it was designed for.
More articles