The Key Rotation That Arrived Too Late: A Crypto Failure in Three Decisions
Archi4 min read
Contents
The Key Rotation That Arrived Too Late: A Crypto Failure in Three Decisions#
On a Tuesday in early March, an engineer at a mid-sized payments processor ran a routine dependency scan and found that a cryptographic library used for token signing carried a flaw in its random number generation. The flaw had shipped two years earlier. The company's signing keys had been generated during that window.
What followed was not a dramatic breach. Nobody exfiltrated customer data. Instead, the company faced a quieter and more expensive problem: it had to decide whether to rotate every signing key in its system, and it had to make that call under deadline pressure from a board meeting it could not move.
The First Decision: Severity Versus Certainty#
The flaw reduced entropy in key generation only in certain environments, mostly containerized deployments where the library fell back to a weaker seed during early boot. The company's production fleet ran in containers. Its staging environment did not. That mattered because staging is where engineers had tested key handling, and where the flaw never appeared.
The security team argued for full rotation. The platform team estimated the work at six to eight weeks, because the signing keys anchored dozens of downstream services, partner integrations, and stored tokens with long expiry windows. Rotating them meant coordinating with external partners who had pinned public keys and could not update on demand.
The company split the difference. It rotated keys for the highest-traffic signing path and deferred the rest to the next quarterly cycle. That decision was defensible on paper. The deferred paths handled lower volume. The exposure window was short. The rollback plan was solid.
What the Deferral Actually Cost#
Three weeks later, a monitoring alert flagged an unusual pattern: a small number of forged tokens validating against a key that should have been retired. The key had been generated in a container image built during the vulnerable window. It had been rotated out of the primary path but stayed live on a legacy endpoint that nobody had inventoried.
The attacker had not broken anything sophisticated. The reduced entropy meant the key space was small enough to search. The forge attempts were cheap to generate and cheap to try.
The company pulled the legacy endpoint within hours. The forensic team found no evidence of data access beyond token validation. But the incident response, partner notifications, and a mandatory security review consumed roughly ten times the engineering hours that full rotation would have required. The deferred work did not disappear. It arrived late, bundled with a crisis.
The Second Decision: How to Rotate#
The company then had to choose a rotation method. Two options were on the table.
The first was a hard cutover: issue new keys, retire old ones on a fixed date, and require every consumer to update. Fast, clean, and brutal for partners who had not built rotation into their systems. Two large partners said they needed at least ninety days. The company had promised regulators a fix within forty-five.
The second was a dual-key period: keep old keys live for validation only, while new tokens signed with new keys. That reduced breakage but extended the window in which forged tokens using the compromised key would still validate.
The company chose a compressed dual-key period, twenty-eight days, with an explicit rule that old keys could validate only tokens issued before the cutover timestamp. That timestamp check turned out to be the detail that saved the program. It meant a forged token with a new issuance date failed even while the old key remained technically active.
The Third Decision: What to Tell Customers#
The disclosure question was harder than the technical one. The company could not prove no tokens were forged during the deferral window. It could prove no customer data was accessed. Legal wanted a narrow statement. The communications lead wanted to get ahead of the story.
They settled on a disclosure that named the flaw, the affected key, the deferral decision, and the timeline. It did not speculate about attacker intent. It did not claim the issue was fully resolved while legacy validation was still active. It committed to a public postmortem within thirty days.
What Carried Over#
The company now runs a key inventory that lists every active key, where it was generated, and what consumes it. That inventory did not exist before the incident. It was the missing piece that made the deferral decision look reasonable at the time and reckless in hindsight.
The second change was a rule about rotation cost estimates. Platform teams now have to state how long a forced rotation would take if it started today, not in a planned cycle. If the answer is longer than the exposure window, the deferral has to be approved above the security team.
The company still runs a dual-key period for every rotation. The difference is that the period now has a fixed end date set before the rotation begins, not negotiated during it. The twenty-eight day window worked. It became the default not because it was optimal, but because it was decided in advance and the timestamp check made it enforceable.
None of these fixes are exotic. The failure was not a failure of cryptography. It was a failure to inventory what the cryptography protected, and a failure to treat the cost of a late rotation as part of the cost of deferring it.
More articles