The Multi-Vendor Router Decision That Left One ISP With a 4-Hour Outage
Archi4 min read
Contents
The upgrade that didn't go to plan#
In late 2025, a regional internet service provider with about 120,000 subscribers faced a decision: standardize on a single router vendor or keep a mix of two. The engineering team chose the mix, citing pricing pressure and a desire to avoid lock-in. Six months later, a routine firmware upgrade on one vendor's edge routers exposed a compatibility gap with the other vendor's core switches. The result was a four-hour outage across three metro areas, affecting roughly 38,000 customers during peak evening hours.
This is an analysis of that decision, not a postmortem of the outage itself. The company has not released a full root-cause report, and details here are based on industry patterns and general vendor documentation. The core question is whether the flexibility of a multi-vendor network was worth the operational risk it introduced.
The case for mixing vendors#
The appeal was straightforward. A single-vendor network means one support contract, one firmware cadence, and one throat to choke. But it also means one price list. The ISP's procurement team estimated that a mixed fleet would save 18 to 22 percent on hardware over five years. That estimate held up in the first year. The savings were real, and the network ran fine.
The problem was not the hardware. It was the software layer between them.
Where the compatibility gap appeared#
Edge routers from Vendor A and core switches from Vendor B both supported the same industry standards on paper. In practice, the standards leave room for interpretation. When Vendor A pushed a firmware update that adjusted how it handled quality-of-service (QoS) marking for voice traffic, Vendor B's switches did not recognize the new marking scheme. Voice packets were treated as best-effort data. Call quality dropped, then failed entirely as retransmissions flooded the links.
The ISP's monitoring tools caught the packet loss but not the cause. Engineers spent the first hour checking fiber links and power. The second hour went to isolating the edge routers. By the third hour, they had rolled back the firmware. Service returned at hour four.
The hidden cost of flexibility#
A single-vendor network would not have avoided every problem. Vendors ship buggy firmware too. But the failure mode would have been different. With one vendor, the QoS change and the switch behavior would have been tested together in the same lab, under the same support contract. The ISP's lab did test the new firmware, but only against Vendor A's own switches. The mixed environment was not replicated because the lab budget did not cover a second full test bed.
That is the trade-off in concrete terms: the savings from mixing vendors came with a testing gap that the ISP did not fund. The 18 to 22 percent hardware savings looked good on a spreadsheet. The outage cost, by conservative estimates, ran into six figures when you count service credits, overtime, and the engineering hours spent on rollback.
What the decision reveals#
The ISP's choice was not reckless. Multi-vendor networks are common and can work well. The mistake was treating the savings as free. Every vendor boundary is a place where assumptions diverge, and those assumptions need testing. The company had a lab, but the lab mirrored only half the network.
A more balanced approach would have been to keep the mixed fleet but fund a small interoperability test bed, perhaps two racks of representative gear from each vendor. That would have added maybe 5 percent to the project cost and closed the gap that caused the outage. The engineering team reportedly proposed exactly that during the procurement review. It was cut in a budget round.
The opinion#
Mixing vendors is a defensible strategy, but only if you pay for the glue. The glue is testing, documentation, and the engineering time to chase down incompatibilities before they hit production. Companies that choose multi-vendor networks to save money and then skip the interoperability work are not saving money. They are deferring a cost that will show up later, usually at the worst possible time.
The ISP's four-hour outage is a useful case because the decision point is so clear. The company had a chance to fund a test bed and chose not to. The firmware upgrade was the trigger, but the missing test bed was the cause. That is a decision other network operators can learn from without waiting for their own outage.
Numbers to keep in mind#
Four hours of downtime for 38,000 customers is roughly 152,000 customer-hours lost. At an average revenue per user of $60 per month, that is about $12,500 in direct service credits, before counting engineering overtime and the cost of the rollback. The hardware savings over five years were projected at several million dollars. The math still favors the mixed fleet, but only if the next upgrade does not repeat the same mistake.
The ISP has since announced a new interoperability lab. The budget line that was cut is back, with a mandate to test every firmware release against both vendors' gear before production. That is the right response. It is also the response that should have come before the outage, not after.
More articles