The Cable Label That Broke a 40G Link: Notes From a Switch Rack Test
Archi4 min read
Contents
The label said A7. The port said A9.#
We spent a Saturday morning chasing a link that would not stay up. Two top-of-rack switches, a 40G QSFP+ trunk between them, and a flapping interface that dropped roughly every 90 seconds. The obvious suspects were optics, firmware, and the DAC cable we had swapped twice. None of them were the problem. The problem was a small printed label on a fiber patch cord, about 3 centimeters wide, that read A7 while the cable actually landed in port A9.
This is a field notes piece about that mislabel, and about the counters that finally told us the truth.
The setup#
The rack held two 32-port 100G switches in a leaf-spine pair, but we were only lighting up 40G ports for this test. A single QSFP+ to 4x10G breakout ran to a storage node. The trunk between the two switches used a pair of 40G QSFP+ SR4 optics over OM4 multimode fiber, roughly 12 meters of it, routed through a cable tray above the rack.
We had built this trunk a dozen times before in other racks. The run was short. The optics were known-good, pulled from a working link in the lab. Power levels looked fine on the light meter. And yet the trunk bounced.
What the interface counters showed#
The first hour was spent on the usual checklist. We reseated both QSFP+ modules. We swapped the breakout on the storage side. We replaced the fiber pair once, then again.
What we did not do early enough was read the error counters carefully. Once we did, the pattern was hard to miss. The interface was not showing CRC errors, which is what you would expect from a dirty connector or a bent fiber. It was showing input discards and a steady trickle of runts, and the link state log showed the trunk alternating between up and down at almost regular intervals.
That combination points to a loop, not a physical layer fault. A physical layer fault gives you CRC errors and light level warnings. A loop gives you discards because the switch is receiving its own traffic back.
The detail that mattered#
We traced both ends of the fiber by hand. The cord labeled A7 on the left switch went into port A9 on the right switch. The cord labeled A9 went into port A7. Someone had crossed the labels during a previous maintenance window, probably when a tech re-patched the tray and put the labels back in the wrong order.
That alone would not cause a loop. What caused the loop was that the previous owner of the rack had left a second fiber pair patched between the same two switches, on ports A7 and A9, and it was still live. So the trunk was being formed twice, once by the intended pair and once by the mislabeled pair, and the switches were seeing each other on two paths with the same port identifiers.
Spanning tree should have caught this. It did not, because the second pair was configured as a member of a static LAG that had one member active and one member down. The switch kept flapping the LAG because the two paths disagreed about which member was primary.
The fix took four minutes#
We pulled the second fiber pair, cleared the static LAG, and let the trunk renegotiate. The interface came up and stayed up. Error counters went to zero within a minute. Total time from the label discovery to a stable link: about four minutes.
Total time spent on the wrong hypotheses: close to six hours.
What we changed in the process#
Three things, none of them expensive.
First, we stopped trusting printed labels. Every fiber pair now gets verified with a light source and a meter before it is patched into a live switch. A 30 second check per pair, done once, prevents this class of problem.
Second, we started reading interface counters before swapping hardware. The counters told us loop within the first ten minutes. We ignored them because the obvious physical suspects were easier to reach.
Third, we wrote a one page pre-flight checklist for any multi-switch trunk test. It lists the four counters to check first (CRC, input discards, runts, link state transitions) and the two configuration items to verify (LAG membership and spanning tree state). It is not clever. It is just written down so we do not skip it at 7 a.m.
The takeaway for anyone running a rack#
The smallest detail in this test was a 3 centimeter label applied in the wrong order during a maintenance window nobody remembered. The cables were fine. The optics were fine. The firmware was fine. The label was wrong, and the leftover fiber from an old design made the wrong label into a loop.
If you are testing a trunk and the link will not settle, check for leftover patching before you check for bad hardware. Then verify every label with a light meter. The extra minute is cheaper than the six hours.
More articles