The Backup Test That Settled a Ransomware Decision
Archi4 min read
Contents
The Night Shift Call#
The call came at 2:17 AM from the operations director of a regional auto parts distributor. Their main server room had been running a ransomware encryption job for at least four hours. The attackers left a note demanding $400,000 in Bitcoin for a decryption key. The company had backups. The question was whether those backups were still trustworthy.
The backup appliance sat on the same virtual LAN as the file servers. That was the first problem. The attackers moved laterally across that network segment for two days before they triggered the encryption. That meant every backup job written during that window was suspect. The company's last known clean snapshot was from eleven days before the attack. My job was to test that snapshot and several later ones in an isolated lab. The decision to pay or restore would rest on what I found.
Building the Test Environment#
I pulled three restore points from the backup appliance. The first was from the night before the attack. The second was from six days before. The third was from the eleven-day-old snapshot.
For each restore point, I built a separate virtual machine on a host with no network route to the production environment. I restored the full system image, then booted each machine in sequence. The goal was to find evidence of the ransomware binary already present in the backup data.
The first restore point showed the attack tooling immediately. A scheduled task named "WindowsUpdate" ran every fifteen minutes. The task executed a PowerShell script from a temp directory. The script had no code signing. The file hash matched the sample from the live incident. That backup was useless.
The second restore point, from six days before, also showed traces. There was an unknown account with domain admin rights. The account had been created four days before the backup. Its creation event was hidden by the attacker, but the backup metadata still recorded the original creation time. That account was present in the backup, so the image was not clean.
The third restore point, eleven days old, was clean. No unknown accounts. No scheduled tasks. No PowerShell scripts in temp. The last file modification on the domain controller was a routine group policy update from four days before that date. This was the restore candidate.
Testing the Restore Path#
A clean snapshot is only useful if you can actually restore it. I tested the full restore procedure in the lab. The backup appliance had its own encryption keys stored on a local disk. The ransomware had encrypted that disk as well. The keys were lost. That forced a re-encryption of the restored data using fresh keys. The restore tool could do this, but it added about six hours to the total time. The company's recovery plan did not account for that step.
I also tested the boot process for each recovered server. The domain controller restored cleanly. The file server restored cleanly. The SQL database server restored cleanly, but the database verification process failed because the backup had been taken during an open transaction. The company had allowed a two-minute window between the backup start and the database freeze. That window was not long enough.
We had to restore the SQL server from a transaction log backup stored on a separate external disk. That disk was disconnected during the attack and remained clean. The log backup added an extra one hour and twenty minutes to the recovery. The total recovery time was 49 hours from the decision point.
The Decision That Mattered#
The operations director had two options. Pay the ransom and receive a decryption key that might work, or restore from the eleven-day-old snapshot and retype or reimport two weeks of customer orders.
The ransom amount was $400,000. The estimated lost order data from those eleven days was around $310,000 in margin. The cost of recovery, including overtime for 30 staff and the external consulting fees, came to another $85,000. Paying the ransom would still require some cleanup work, because the decryption tool would not remove the attacker's backdoor account or the scheduled task. The total cost of paying was roughly $500,000.
The restore path cost about $395,000 in real terms, but it also meant the company would not have to trust a decryption tool from an attacker. The decision was made to restore.
During the restore, I found one more problem. The backup appliance's administrative password was the same as the domain admin password. That password appeared in the attacker's credential dump. The company had to change every administrative password in the environment before the restored servers could join the network. That took another three hours.
This incident came down to a single question: could we trust the backups? The answer took two days to establish. The company's recovery plan assumed backups were clean by default. That assumption failed. The decision to restore instead of pay was based on test results, not faith. The clean snapshot was old, but it was verifiably clean. The restore path was tested before committing to it. The company accepted the data loss because the alternative carried more risk.
We documented everything in the field notes, including the time stamps of each test. The operations director kept a copy for the cyber insurance claim. The insurer required proof that the backups were tested and that the decision was based on evidence. That proof came from the lab restores, not from the marketing sheet of the backup vendor.
The company now runs a monthly restore test on a separate network segment. The backup appliance credentials are no longer the same as the domain admin. The backup network is separated by a firewall rule. Those changes came after the decision, not before it. That is the common pattern in these incidents. The decision point exposes what the recovery plan actually contains.
More articles