A Misconfigured Router, 70 Minutes of Detour, and a Very Different Internet
Archi3 min read
Contents
Two Routers, One Policy Change, and 70 Minutes of Wrong Turns#
At 09:14 UTC on Tuesday, a regional transit provider pushed a policy change to two border routers. Within four minutes, traffic bound for a large cloud region in Western Europe was being handed to a network with no capacity to carry it. The detour ran about 70 minutes.
The symptoms were ordinary, which is why they took time to read. Checkout pages stalled during the TLS handshake. Mobile apps that call three APIs per screen failed on the second call. A voice carrier reported one-way audio on calls between two cities 400 km apart, which looked like a codec problem and was not.
What a route leak actually does#
BGP is a rumor mill with good manners. Each network announces the addresses it can reach and the path it will use, and every neighbor takes the claim largely on trust. The base protocol does not check whether the announcing network is authorized, and it does not check whether the path is one the sender should be using.
A leak happens when a network re-announces routes it learned from one provider back out to another. Usually a filter is missing, or a policy is applied in the wrong order. It is not an attack. It is a typo with a blast radius.
The distinction from a hijack matters. In a hijack, someone claims address space they do not own. In a leak, the origin is correct and the path is wrong. Origin validation catches the first case. It does nothing for the second.
Five years ago, the same mistake traveled further#
In 2021, roughly half of the IPv4 address space in the global table sat behind a signed Route Origin Authorization. Public trackers now put coverage well past two thirds, and the largest prefixes are higher still. The more important shift is enforcement: a growing share of large transit networks actually validate those signatures and drop invalid announcements at the edge.
The protocol has not changed. The number of networks willing to act on it has.
Anycast absorbed part of Tuesday's damage. Five years ago a regional routing fault often meant a hard outage for a DNS resolver or a CDN edge. Today the same failure reroutes to the next point of presence, and the user sees a slow page rather than an error.
The monitoring gap that closed#
Polling routers every five minutes over SNMP was standard in 2021. A 70-minute event could hide inside two poll cycles, so engineers learned about it from customers. Streaming telemetry, now shipped by default on most modern chassis, pushes prefix counts and interface state every few seconds. It did not prevent Tuesday's leak. It cut the gap between the first symptom and the right diagnosis, and that gap is where most outage cost sits.
Who pays for the detour#
Retry logic decides the bill. A client with a 30-second timeout and three retries turns a short routing fault into 90 seconds of occupied connections. Mobile operators with aggressive connection reuse saw queues build on their own side, because half-open sessions to the affected region kept the radio awake.
Cloud status pages reported elevated error rates for about an hour. Merchants saw abandoned carts. The transit provider saw a routing table that looked fine, because every path in it was valid.
Where the safety net still has holes#
Authorization is not path validation. A network can hold a valid ROA, announce from the correct origin, and still push traffic through a provider that should never see it. The standard that addresses this, ASPA, is in early deployment. Until it is widely signed and enforced at the edges, leaks will keep arriving with correct origins and wrong paths.
Smaller networks remain exposed. Signing an ROA costs nothing but time, and time is what a five-person ISP does not have. Enterprise edges are uneven in the other direction: many organizations validate routes coming in from transit and never check what their own routers announce outward.
What to watch over the next five years#
The internet got better this week at rejecting false claims about ownership. It stayed roughly as bad at rejecting false claims about paths. Tuesday's 70 minutes were shorter than the 2021 equivalent would have been, and still long enough to cost real money in carts and calls.
The next threshold is path validation. Watch which transit providers publish ASPA objects and which publish nothing. That silence says more about your failover plan than any SLA document does.
More articles