The Stale DNS Record That Turned a Security Audit Into a Fire Drill
Archi5 min read
Contents
- The record nobody owned
- What the audit caught and what it missed
- The six hours that mattered more than the six months
- Hour one: stop the bleeding without breaking payroll
- Hours two through four: the question nobody wanted to ask
- Hours five and six: the paper trail
- The smallest detail is usually an ownership gap
- What I would tell the next team
The record nobody owned#
Incident reviews love the dramatic moment: the phishing email that landed, the credentials that leaked, the ransomware note on a printer. The detail that decides how bad a breach gets is usually smaller and duller. In a review I followed last year, it was one DNS record: a wildcard A entry pointing at an internal load balancer that had been decommissioned 14 months earlier. Nobody deleted it because nobody knew who owned it.
The wildcard did what wildcards do. Any subdomain that did not have its own record resolved to that address. When the balancer was replaced, the new host still answered on the same private IP range, so most traffic kept working. A handful of internal tools quietly started talking to the wrong place. One of them was a log shipper authenticated with a service account that had read access to a production database.
That is the whole incident. No zero-day, no nation-state, no clever malware. A routing table from 2021 that outlived its purpose.
What the audit caught and what it missed#
The external assessment found the stale record in about four hours. This is the part that gets lost when people talk about detection. Finding the problem was not the hard part. The auditors had a scanner, the scanner flagged an orphaned record, and the report listed it as medium severity. The finding sat in a queue for 11 days because the team that owned DNS had been folded into platform engineering during a reorg, and the ticket routing rules still pointed at a mailbox nobody read.
By the time an engineer picked it up, the service account had been used from three unfamiliar internal hosts over a six-week window. Not exfiltration on a Hollywood scale. A few thousand rows a night, the kind of volume that hides inside normal backup and reporting traffic.
Here is my view, and it is not a popular one in security circles: the audit worked. The tooling did its job. What failed was the handoff between finding a problem and giving it to a human with the authority to fix it. Companies spend enormous sums on detection and comparatively little on the boring plumbing that decides whether a finding becomes a fix or a line in a PDF.
The six hours that mattered more than the six months#
The team found out on a Tuesday morning. The wildcard record was removed by 11 a.m. The service account was rotated by lunch. Then the real work started, and this is where most organizations either contain a problem or let it grow.
Hour one: stop the bleeding without breaking payroll#
Removing a wildcard in a live environment is a gamble. Automated jobs, monitoring probes, and a surprising number of one-off scripts depend on names that resolve through it. The team ran a query against DNS logs from the previous 30 days and found 41 distinct hostnames hitting the wildcard. Most were decommissioned projects. Two were live: a payroll integration and a facilities badge reader sync.
The temptation in that moment is to rip out the record and apologize later. The engineer on call did the opposite. They narrowed the wildcard to the two live hostnames, kept the record for everything else pointed at a black hole, and monitored for breakage. Payroll never noticed. That decision, made in about 20 minutes, is worth more than most incident response playbooks I have read.
Hours two through four: the question nobody wanted to ask#
The harder call was whether to treat the service account as compromised. The logs showed access from unfamiliar hosts, but the hosts were in a subnet used by a contractor team. The account had no MFA because it was a machine identity, and the rotation policy had been set to 365 days by someone who left the company before the policy shipped.
My opinion: when a credential has been reachable through an unintended path for an unknown period, the default should be rotation, not investigation. Investigating first feels rigorous. In practice it buys the attacker time and burns analyst hours on a question that may never resolve cleanly. Rotate, then investigate. The team did this at hour three, and the log noise stopped within minutes.
Hours five and six: the paper trail#
The last stretch was legal and communications. The database rows that left the network were customer records, mostly names and contact details, plus a slice of order history. Whether that triggers a notification obligation depends on jurisdiction and on whether the data was encrypted at rest in a way that makes it unreadable. It was not. The team spent two hours confirming that, and the answer shaped everything that followed.
This is the detail I keep coming back to. The breach did not become a legal event until someone confirmed the encryption posture of a database column that had been added by a contractor in 2022. That confirmation took longer than removing the DNS record, rotating the credential, and containing the whole thing combined.
The smallest detail is usually an ownership gap#
The wildcard record is a symptom. The disease is that nobody could answer the question "who owns this?" without a meeting. Reorgs, contractor turnover, and the quiet departure of the one person who understood the old setup all create records, accounts, and certificates that outlive their purpose and their owners.
A few things that would have changed the outcome, none of them expensive: , A quarterly report of DNS records that have not received traffic in 90 days, routed to a named owner rather than a team alias, Service account rotation tied to a calendar event, not a policy document, A rule that any finding from an external audit gets assigned to an individual within 48 hours or escalates to a director
The last one is the one I would fight for. Severity scores tell you how bad a finding looks. Assignment tells you whether it will ever be fixed. A medium-severity record with a named owner beats a critical finding in a shared queue every time.
What I would tell the next team#
Do not read this as a story about DNS. Read it as a story about the gap between knowing and acting. Most organizations can find their stale records. Far fewer can name the person who will delete them, and almost none can do it within a day without breaking something. The team in this incident did the hard part well: they contained a small problem before it grew, and they made the risky call to rotate first and investigate second.
The record is gone. The reorg is not. Somewhere in your environment, a wildcard is pointing at a server nobody remembers, and the ticket is sitting in a mailbox that stopped being read in March.
More articles