A Lost Signing Key Took Down 4,000 Stores. Compare That to 2020.
Archi4 min read
Contents
The Failure, in Order#
The terminal fleet started failing at 06:12 on a Tuesday. Store staff saw a spinner where a payment confirmation should have been. By 06:40 the help desk had 1,100 tickets. By 08:00, roughly 4,000 stores across three countries could not complete a card transaction on the primary terminal.
The cause was a code-signing certificate. An intermediate certificate in the chain had expired at midnight. Nobody had renewed it because the renewal job pointed at a mailbox that had been renamed during a reorganisation eighteen months earlier, and the reminder emails bounced silently into a distribution group with no members. The certificate itself had a five-year term. It had been issued in 2020.
Recovery took nine hours and four minutes. The new certificate was issued in ninety minutes. Getting it onto 4,000 terminals that only accept signed packages, over a rollout channel that itself needed a signed package, took the rest of the day.
What Changed Since 2020#
Five years ago, this company ran the same fleet with a much simpler estate. Terminals pulled updates from one region. The certificate chain had two links. A single team owned the whole path from build to deployment, and that team sat in one room.
Now the chain has five links. The build runs in a hosted pipeline. Artefacts land in a third-party registry. Two acquisition targets brought their own intermediate certificate authorities, each with different lifetime policies. The rollout service is managed. Each handoff added a trust relationship that nobody drew on one diagram.
That is the pattern behind most of the security failures I have looked at this year. The individual controls are stronger than they were in 2020. Password hygiene is better. Disk encryption is near universal. The problem has moved to the seams between systems, and there are simply more seams.
Three numbers make the shift concrete. Ten years ago, a mid-sized retailer might run 40 internet-facing services. This one runs more than 400. The average certificate lifetime at this company dropped from three years to about thirteen months, which is good practice, but it triples the number of renewal events per year. And the number of people who can approve a change to the signing chain went from four to twenty-two after the acquisitions.
More short-lived credentials, more approvers, more handoffs. Each change is reasonable on its own. Together they raise the odds that one renewal falls through a gap.
The Controls That Did Not Help#
A key management system flagged the expiry ninety days out. The alert went to the renamed mailbox. A monitoring dashboard showed certificate age in a panel that had been scoped to the primary region only, so the two acquired brands never appeared.
Both controls worked as configured. Neither was configured against the estate as it actually existed.
This is the difference from five years ago. In 2020, a broken alert usually meant a broken control. In 2026, a broken alert often means the control was pointed at the wrong inventory. The inventory problem is harder. It requires someone to know every place a certificate, key, or token is trusted, and that knowledge decays the moment a team reorganises.
What the Postmortem Got Right#
The incident review, published eleven days later, did three things well.
First, it counted business impact in transactions, not systems. The headline was 61,000 failed card attempts, not "4,000 terminals affected." The second number sounds like an inventory problem. The first sounds like money.
Second, it traced the mailbox rename to a specific ticket from March 2024 and named the process gap without naming individuals. That detail matters because the same reorganisation pattern exists across the company, and other teams can now check their own alert routing before the next audit finds it for them.
Third, it shortened the certificate term on the internal authorities to ninety days. Shorter lifetimes are counterintuitive after an expiry failure. The reasoning is sound: a renewal that fails every quarter gets noticed. A renewal that fails once every five years does not.
What It Got Wrong#
The review stopped at the certificate. It did not ask why the rollout channel needed a signed package to deliver a signed package, which is the reason nine hours turned into a full day. It also did not ask how many other services depend on the same mailbox routing, which is the question that would have prevented the next incident rather than explaining this one.
And it set an action item to "improve certificate visibility" without specifying what visible means. That item will be closed in six months with a new dashboard. The dashboard will be scoped to whatever inventory someone remembers to feed it.
The Comparison Worth Making#
In 2020, an expired certificate was a comedy. Teams joked about it because the fix was usually a single server and a single deploy. The joke has aged badly.
The estate is now large enough that a routine expiry can halt revenue in three countries, and the recovery time is set by how many signed handoffs sit between the certificate authority and the terminal. Counting those handoffs is a more useful exercise than counting certificates. The first number tells you how bad the bad day will be. The second only tells you how many bad days are queued up.
A reasonable next step for any team running a fleet this size: take the last three reorganisation tickets and check where every monitoring alert was routed before and after. The answers will be uncomfortable. They are cheaper to find now than at 06:12 on a Tuesday.
More articles