The Idempotency Key Miss That Duplicated 9,400 Payments, and Why It Would Have Been Caught in 2020

Archi4 min read

paymentsfailurerisk

Contents
  1. The duplicate charge that took six hours to find
  2. What the retry path looked like five years ago
  3. Three things that changed
  4. Why the check was missing
  5. What to check now

The duplicate charge that took six hours to find#

On a Tuesday morning in late March, a mid-sized payment processor pushed a routine update to its card authorization service. Within 40 minutes, 9,400 cardholders saw two identical charges on their statements. The fix took six hours to deploy. Refunds took nine days.

The root cause was a missing idempotency check on a retry path. When the acquirer timed out, the gateway retried the authorization, and nothing on the processor side recognized the second request as the same transaction. The first charge settled. The second charge also settled. Merchants saw double payouts, then double chargebacks.

Five years ago, this failure mode was less likely to reach customers. Not because engineers were more careful, but because the surrounding systems were slower, simpler, and less connected. The gap between what a retry looked like in 2020 and what it looks like today is the actual lesson.

What the retry path looked like five years ago#

In 2020, most card-not-present traffic on this processor moved through a single authorization endpoint with a 30-second timeout and a fixed retry budget of two attempts. The ledger wrote a settlement record on every successful auth response. Duplicates happened, but they were rare enough that a nightly reconciliation job caught most of them before settlement.

That reconciliation job ran at 2 a.m. and compared the day's authorizations against the day's captures. A mismatch generated a ticket. A human resolved it. The median time to resolution was 11 hours. Nobody called this reliable, but it worked because the volume was manageable and the failure was visible in a batch report by morning.

Today the same processor runs three authorization endpoints (one legacy, one for tokenized cards, one for real-time payments), each with its own retry semantics. The reconciliation job now runs hourly on a streaming pipeline. It still catches duplicates, but only after the authorization has already been sent to the network. By then, the cardholder has a pending charge.

Three things that changed#

The first change is request volume per second. The processor's peak authorization rate in 2020 was around 1,200 per second. In March it was 8,900 per second. A retry loop that once produced a few hundred duplicates per hour now produces thousands per minute. The idempotency check was never fully implemented on the new tokenized endpoint because the initial design assumed the legacy endpoint's retry budget applied. It did not.

The second change is the number of hops between the merchant and the issuer. Five years ago, a typical transaction crossed four systems: merchant, gateway, processor, network. Now it crosses seven, including a real-time fraud scoring service and a token vault. Each hop can retry independently. Without a shared idempotency key that survives every hop, the same logical payment can arrive at the network more than once.

The third change is settlement timing. Real-time payments settle in seconds. Card authorizations still batch, but the processor added an instant capture option for small merchants. That option writes to the ledger before the nightly reconciliation runs. In 2020, the batch window gave engineers a safety net. In 2026, the safety net is gone for a growing share of volume.

Why the check was missing#

The idempotency key existed in the API spec. It did not exist in the code path that handled acquirer timeouts. A code review in February flagged the gap, but the fix was scheduled for the next quarter because the team prioritized a different migration. The risk register listed duplicate charges as a medium-severity issue with a low likelihood. That rating was based on 2020 traffic patterns.

This is the pattern to watch. Teams rate risk using the system they remember. When volume triples and hops multiply, the likelihood changes without anyone updating the register.

What to check now#

Start with the retry budget on every endpoint that can trigger a settlement. If two endpoints share a merchant and a card but not an idempotency key, they can duplicate. Map every hop between the merchant and the network and confirm the key is passed through unchanged. Then measure the time between authorization and reconciliation. If that window is shorter than your mean time to detect a duplicate, the batch job is no longer a control.

The processor in this case has since added a deduplication service that sits in front of all three endpoints and rejects any authorization with a key seen in the prior 24 hours. It cost less than the refunds. The harder fix is the risk register, which now includes a quarterly review of traffic assumptions. That review is the part most teams skip, and it is the part that would have caught this five years earlier.

More articles