How a Single Misconfigured S3 Bucket Led to a Mass Data Exposure
Archi4 min read
Contents
The Bucket That Shouldn't Have Been Public#
On a Tuesday morning in March, a security researcher found a publicly listable Amazon S3 bucket belonging to a mid-sized healthcare analytics firm. Inside were 120,000 records: names, dates of birth, insurance member IDs, and partial claims data. The bucket had been private for two years. Then, in late January, an engineer made a single permission change while debugging a data pipeline. That change went unnoticed for 47 days.
The breach made headlines for a day. The more useful story is technical: how one API call removed the last line of defense, and why nothing in the firm's tooling caught it.
The Change That Started It All#
The engineer was troubleshooting a stalled ETL job. A downstream service was failing to read objects from the bucket, throwing 403 errors. Under time pressure, the engineer ran a command to grant public read access to the bucket's ACL, intending to revert it after testing. The command succeeded. The test passed. The revert never happened.
In AWS, an S3 bucket's public access is governed by several layers: bucket policies, ACLs, and account-level block public access settings. The firm had block public access enabled at the account level, which should have overridden the ACL change. But three weeks earlier, a different team had disabled that account-level setting to support a legitimate public-facing website hosted in the same account. The setting was never re-enabled.
So the ACL change took effect. The bucket became publicly listable. No alert fired.
Why the Safety Nets Failed#
The firm had logging enabled, but not the right kind. Server access logs were turned on, capturing every request to the bucket. However, no one was monitoring for GetBucketAcl or PutBucketAcl calls, and the logs were retained for only seven days. By the time the researcher's report arrived, the evidence of the original change had expired.
CloudTrail was active, but the firm had not configured any alarms on management events for S3 permission changes. The security team relied on a weekly manual review of IAM changes, which focused on user and role modifications, not bucket-level policies.
A third-party cloud security posture tool was in place, but its S3 checks were limited to encryption and versioning. Public access was not in its rule set because the tool's default policy had been customized months earlier to reduce alert noise. The customization removed the very check that would have flagged the bucket.
The Root Cause in Technical Terms#
The failure was not a sophisticated attack. It was a sequence of three independent decisions that, combined, removed all guardrails:
- Account-level block public access was disabled for a legitimate use case and not restored.
- An engineer granted public read access to a bucket without a change management ticket or peer review.
- Monitoring and alerting did not cover S3 permission changes, and log retention was too short to investigate after the fact.
Each decision seemed reasonable in isolation. Together, they created a window where a single API call could expose sensitive data. The firm's architecture had multiple layers of defense on paper, but those layers were not independent. Disabling one disabled the others.
What the Incident Response Revealed#
The firm's detection came from outside: a researcher's email to a generic security address. Internal tools did not detect the exposure. The response team took 14 hours to confirm the bucket was public, partly because no one on the on-call rotation had experience with S3 ACLs. The bucket was made private within 24 hours of the report, but the data had been publicly listable for 47 days.
Forensic analysis was hampered by the seven-day log retention. The team could not determine whether anyone had actually downloaded the data, only that the bucket had been listable. The firm notified affected individuals as a precaution.
Lessons for Engineers and Architects#
Treat permission changes as production changes. The engineer who modified the ACL did not go through change management because the action felt like a test. In cloud environments, permission changes are production changes. They should require the same review and rollback plan as a code deploy.
Make guardrails independent. Account-level block public access is a powerful control, but it can be disabled by anyone with sufficient permissions. Pair it with service control policies that prevent disabling the block, and with automated remediation that re-enables it if changed.
Monitor the control plane, not just the data plane. Access logs show who read objects. CloudTrail shows who changed permissions. The latter is more important for detecting exposure. Alert on PutBucketAcl, PutBucketPolicy, and DeletePublicAccessBlock events, and route them to a channel someone actually watches.
Retain logs long enough to investigate. Seven days is not enough. Many breaches are discovered weeks or months after they begin. A 90-day retention for management events is a reasonable minimum.
Test your detection. The firm's third-party tool would have caught the issue if its default rules had been left intact. Regular tabletop exercises that include cloud permission changes can reveal gaps before an attacker does.
The breach was not caused by a zero-day or a clever attacker. It was caused by a configuration change that no one saw, in an environment where the safety nets had been quietly removed. The fix is not more tools, but more discipline around the ones already in place.
More articles