Field Notes: Choosing Disconnection Over a 30-Day Breach Clock
Archi3 min read
Contents
The call at 3 a.m.#
A mid-sized logistics firm, which I will call Meridian Freight, had 48 hours to decide whether to pay a ransom after attackers encrypted its billing system and stole 1.2 TB of customer data. I was on the incident response team. My job was to test the backup chain and tell the board how fast we could rebuild if they chose not to pay. The answer turned out to be 30 days. That number changed the entire conversation.
What the backups actually held#
The company ran nightly backups to a local NAS and weekly copies to a cloud bucket. On paper, that is a reasonable setup. In practice, the NAS had been reachable from the same flat network as the billing servers. The attackers did not encrypt it, but they did delete the last 14 daily snapshots. The cloud bucket was versioned, which saved us, but the versioning had been enabled only eight months earlier. Anything older was gone.
I tested a restore on an isolated VLAN. It took 11 hours to pull the full dataset and another 6 to verify checksums. That was the good news. The bad news was the application layer: the billing software kept its own audit logs on the compromised servers, and the vendor could not reissue them. We could restore data, but we could not restore the legal record of who changed what and when.
The three options on the table#
By the second day, the board had three choices. Pay the ransom (the demand was 90 bitcoin, roughly $5.8 million at the time) and hope for a decryptor. Rebuild from the eight-month-old cloud snapshot and accept the data loss. Or rebuild forward from the current state using the surviving cloud versions plus manual reconciliation from paper invoices and bank statements.
The CFO pushed for payment. The general counsel pushed back hard: paying a ransom to a sanctioned group can create its own legal exposure, and the stolen data was already out. The CEO asked me one question. How long until we can bill customers again?
What the restore test showed#
I ran the rebuild-forward scenario over a weekend. We restored the cloud bucket to a staging environment and then spent 19 hours writing scripts to reconcile invoices against bank feeds. About 4% of records did not match. Those needed human review. Two staff members spent three days on 1,800 records. It was tedious, but it worked.
The eight-month-old snapshot was faster to restore but far worse in practice. It would have meant telling customers their recent payments were not recorded. The reputational cost of that, the general counsel argued, was higher than the ransom.
The decision and the cost#
Meridian chose the rebuild-forward path. No ransom. The final bill: $740,000 in incident response and legal fees, $210,000 in overtime and temporary staff, and a 9-day gap in billing. The company notified regulators within 72 hours as required, and the breach notification to customers went out on day 6. The CEO later told me the 30-day rebuild estimate was what tipped the board. It gave them a worst case they could survive, which made the ransom unnecessary.
What I would test next time#
Three things stood out. First, backup versioning is only useful if you know when it started. If versioning is 8 months old, your real recovery point is 8 months, not 30 days. Second, application logs need to live somewhere the attacker cannot reach. Meridian now ships audit logs to a separate account with write-only access. Third, the reconciliation scripts we wrote under pressure should have been written earlier, as a tabletop exercise. A restore test that does not include the business logic is only half a test.
The ransomware group posted a sample of the stolen data on its leak site two weeks later. Meridian's customers got a lot of emails. None of them asked about the ransom. They asked when their invoices would be correct. That is the part the board remembers.
More articles