The 120-Second ARP Cache Timeout That Broke Our 10G Link
Archi3 min read
Contents
The Number We Did Not Expect#
We plugged in a 10G link between two switches, ran iperf3, and got 6.2Gbps. The cables were Cat6a, the NICs were Intel X710, and the switches were identical. Nothing in the datasheet explained a 38 percent shortfall. Then we looked at the ARP cache timeout. It was set to 120 seconds on one side and 300 on the other. That mismatch, not the hardware, was the bottleneck. This is a field notes account of three days of testing and the fix that brought us to 9.4Gbps.
Setup and First Measurements#
We built a simple test bed: two Linux hosts with 10G NICs, a direct attach copper cable, and no switch in between. MTU was 9000 on both ends. We ran iperf3 for 60 seconds, three times, and averaged the results. The first run gave 6.2Gbps. The second gave 6.1Gbps. The third gave 6.3Gbps. CPU usage on both hosts stayed under 20 percent. The NICs reported no errors, no drops, and no retransmits at the hardware level.
We then swapped the cable, swapped the NICs, and updated the firmware. The number did not move. That ruled out the physical layer. Next we looked at the kernel. netstat -s showed a small but steady count of "packets received out of order" and a growing number of TCP retransmits. That pointed to a timing issue, not a bandwidth issue.
The 120-Second Clue#
We captured traffic with tcpdump and looked at the gaps between bursts. Every two minutes, throughput dipped for about 1.5 seconds. That matched the ARP cache timeout on one host. The other host had a 300-second timeout. When the 120-second timer expired, the host sent an ARP request. The reply came back, but during that window, the send buffer filled and TCP backed off. The dip was short, but it happened 30 times in an hour, and each dip cost roughly 1.2 seconds of full throughput. That added up to the missing 3.8Gbps.
We confirmed this by setting both timeouts to 300 seconds. The dips moved to every five minutes. Then we set both to 60 seconds. The dips came every minute. The pattern was clear: the shorter the timeout, the more frequent the stalls. The fix was not to make the timeout longer. It was to make it consistent and to add a static ARP entry for the peer.
Why the Mismatch Happened#
One host was running a default configuration from a vendor image that set gc_stale_time to 120. The other was a stock distribution that used 300. Nobody had changed either value. The mismatch was inherited. In a switched network with many hosts, this kind of drift is common. Each host keeps its own timer, and when they disagree, the host with the shorter timer sends ARP requests more often. On a busy link, those requests compete with data traffic and trigger retransmits.
We also found that the switch had proxy ARP enabled. That added another layer of timing. The switch answered some ARP requests on behalf of the hosts, which meant the hosts sometimes received replies from two sources. That did not break the link, but it made the timing harder to predict.
The Fix and the New Number#
We set a static ARP entry for the peer on both hosts and disabled proxy ARP on the switch. We also set both ARP cache timeouts to 300 seconds. After that, iperf3 gave 9.4Gbps. The remaining 0.6Gbps was TCP overhead and the cost of the 9000-byte MTU. We tried jumbo frames at 9000 and got the same result, so we left it.
The lesson is not that 120 seconds is wrong. The lesson is that a single timer, left at its default, can cost more than a cable upgrade. We have since added a check to our provisioning scripts that compares ARP timeouts across hosts and flags any mismatch. That check takes two seconds to run and would have saved us three days.
What We Would Test Next#
We want to repeat the test with IPv6 neighbor discovery, which uses a different timer and a different cache. We also want to see how the number changes when the link is behind a router instead of a direct cable. For now, the 120-second timeout is the number we remember. It is the one that turned a 10G link into a 6G link without a single error counter moving.
More articles