The 40 Millisecond Fix: What a Fiber Cut Taught Us About Failover

Archi4 min read

networkingtelecomoutagereliability

Contents
  1. The Number Was 40 Milliseconds
  2. What Actually Broke
  3. The Checks That Failed Before the Cut
  4. The Surprising Part
  5. What the Drill Data Showed
  6. The Backup Path Is a Product, Not a Spare
  7. What to Take From This

The Number Was 40 Milliseconds#

The outage report landed on a Tuesday. A contractor trenching for a new subdivision sliced a 96-strand fiber bundle at 1:14 p.m. The bundle carried traffic for a regional ISP's eastern ring, about 11,000 residential customers and 340 business accounts. Service dropped for some customers. For most, the gap lasted roughly 40 milliseconds, the time it took for routers to detect the dead link and shift traffic to a microwave backup path.

Forty milliseconds is not impressive in a lab. It is remarkable in the field, because the failover almost did not happen. The backup path had been provisioned 14 months earlier and had failed three separate readiness checks before anyone trusted it. The story of that 40 milliseconds is mostly a story about the 14 months.

What Actually Broke#

The fiber cut was the easy part. The hard part was what the failover exposed. The microwave link had been sized for average evening load, not for the burst that arrives when 11,000 connections re-handshake at once. For about six seconds after the switch, the backup path ran at 94 percent utilization. Latency for voice customers spiked to 300 milliseconds. A handful of VoIP calls dropped outright.

The ISP's network lead put it plainly in the incident review: the failover worked, and the experience was still bad. Those are different measurements, and only one of them shows up in a dashboard that tracks link state.

The Checks That Failed Before the Cut#

Three readiness tests over 14 months had each found something: , A routing policy that preferred the primary path even when it was down, so traffic would have black-holed instead of failing over., A stale ARP table on one aggregation router that delayed convergence by roughly two seconds., A backup link with no committed bandwidth, meaning the carrier could deprioritize it during congestion.

None of these showed up in normal monitoring. The link was up. Latency was fine. Utilization was low. The configuration was wrong in ways that only matter when the primary path disappears, and nobody tests for that on a random Thursday.

The Surprising Part#

The 40 milliseconds was not the result of better hardware. It came from one change: the team started running scheduled failover drills, one per month, during business hours, with a rollback plan and a 15-minute window. The first drill caused a 90-second outage for 400 customers. The fourth drill caused a four-second blip. By the seventh, customers did not notice.

That is the lesson, and it is uncomfortable. The number that looks good in the report came from deliberately breaking things on purpose, repeatedly, while people were watching. Most network teams know this. Far fewer do it, because a drill is an outage you scheduled, and scheduled outages are hard to defend in a change review.

What the Drill Data Showed#

After eight months of drills, the team had a table of real convergence times per path, per time of day, per traffic mix. The numbers were not flattering. The primary-to-backup switch averaged 40 milliseconds. The backup-to-primary switch back averaged 220 milliseconds, because the microwave path had to be drained and the carrier's routing had to age out stale entries.

That asymmetry matters. A brief cut is survivable. A cut followed by a slow, flapping return can be worse than either state alone, because applications retry, sessions reset, and users see intermittent failures that look like a broken service rather than a brief outage.

The team fixed the return path by shortening the carrier's hold-down timer and pre-building the primary route so it did not have to be relearned from scratch. The return time dropped to about 60 milliseconds. That change came from the drills, not from a vendor's reference architecture.

The Backup Path Is a Product, Not a Spare#

One more finding is worth repeating to anyone who runs a network with a redundant link. The microwave backup had been treated as a spare, kept online, rarely examined. In practice it was carrying production traffic for part of every month because of unrelated maintenance on the primary. It had been doing real work with no real monitoring.

After the incident, the ISP put the backup path into the same monitoring, capacity planning, and change review as the primary. The microwave link got a committed bandwidth clause in its contract. The team started tracking convergence time as a first-class metric, charted next to uptime, because uptime alone cannot describe what customers feel during a cut.

What to Take From This#

The 40 millisecond number is real, and it is the kind of figure that makes a good slide. The useful part is the sequence behind it: three failed readiness checks, eight scheduled drills, one uncomfortable 90-second outage during the first drill, and a slow return path that only showed up because someone measured both directions.

If there is a transferable rule, it is this. Measure the failover, not just the failure. A link that goes down is easy to see. A link that comes back slowly, or comes back on a path nobody sized, is the one that turns a 40 millisecond event into an afternoon of support tickets. The drill schedule is what closes the gap between those two outcomes, and it is almost always cheaper than the incident review that follows the alternative.

More articles