The Deletion Bug That Left a Year of User Data in Cold Storage
Archi3 min read
Contents
The Root Cause Was a Deleted Row That Was Never Deleted#
When a mid-sized photo storage service told users their account deletion requests were complete, the company believed it. The privacy policy said data was removed within 30 days. A support dashboard showed green checkmarks. Then a routine data mapping exercise for a new regulator inquiry found the truth: photos, captions, and location tags for roughly 18,000 closed accounts were still sitting in an object storage bucket in a secondary region.
The failure was not a policy failure. It was a foreign key.
The company's deletion service had been built two years earlier around a simple pattern. A user requests deletion. A background job marks the account row as deleted in the primary database. Cascading deletes remove profile records, session tokens, and payment methods. The job writes an audit log entry and sends a confirmation email. From the application's perspective, the user is gone.
What the job never touched was the media table in a separate shard. That table held a reference to the account ID but no foreign key constraint back to the users table. The schema had been split during a scaling effort, and the constraint was dropped to reduce lock contention during bulk inserts. The deletion job was never updated to compensate.
Why Nobody Caught It for Eleven Months#
The bug survived because every test passed. Test environments used synthetic accounts with no media attached, so the orphaned row path never triggered. The audit log recorded the job as successful because the job did complete its defined steps. The confirmation email was accurate about the account record and wrong about everything else.
The monitoring was also misleading. The team tracked deletion job completion rates, which stayed above 99.8 percent. They did not track post-deletion data retention by table. A dashboard showing rows removed per table would have exposed the gap in a week.
This pattern is common in privacy engineering. Deletion is treated as an application state change rather than a data lifecycle problem. Regulators, including those enforcing the GDPR and several U.S. state privacy laws, treat deletion as an outcome: the data must actually be gone, not just hidden from the interface.
The Soft Delete Trap#
The service also used soft deletes. Account rows were flagged with a deleted_at timestamp instead of being physically removed. That is a reasonable design for recovery windows and for legal holds. It becomes a liability when teams forget that soft-deleted rows still count as retained personal data.
In this case, soft-deleted accounts stayed in the primary database for 90 days before a hard delete job ran. That job also skipped the media shard. So the data had two independent paths to survival, and both were open.
A data protection impact assessment completed a year earlier had listed the media shard as out of scope because it was described as a content delivery cache. It was not a cache. It was the primary store for original uploads.
What the Fix Required#
The remediation took nine weeks and touched more systems than the original deletion feature. The team added a foreign key with an ON DELETE CASCADE constraint to the media table, then ran a one-time backfill to identify orphaned rows. They wrote a deletion coordinator service that fans out to every data store, including object storage, search indexes, and analytics pipelines, and waits for acknowledgments before marking a request complete.
They also added a weekly reconciliation report that compares deleted account IDs against row counts in each downstream store. Any mismatch above a threshold pages the on-call engineer. The report is boring, which is the point.
Lessons for Teams Building Deletion Flows#
The technical lessons are not exotic. Map every store where personal data lands, including backups, logs, and third-party processors. Enforce referential integrity or write explicit cleanup logic for every shard. Treat soft deletes as retention, not removal. Test deletion with realistic data volumes and attached media, not empty accounts.
The organizational lesson is harder. Deletion is a cross-team contract. The team that owns the account database rarely owns the media pipeline, the search cluster, or the analytics warehouse. Someone has to own the end-to-end outcome and verify it with evidence, not with a green checkmark on a dashboard.
The company disclosed the issue to regulators within the required window, notified affected users, and extended its deletion timeline to 45 days while the new coordinator was tested. The fine, if any, will be decided later. The engineering bill arrived first.
More articles