The Missing Comma That Broke a Package Registry for Two Days

Archi4 min read

open-sourcesoftwareyamlreliability

Contents
  1. The detail nobody caught in review
  2. The release that went out on a Friday
  3. The comma, not the parser
  4. The cost and the change
  5. One more thing about the fix

The detail nobody caught in review#

In open source, the failures that teach the most are rarely the dramatic ones. They are the small ones that slip through because everyone was looking somewhere else. This interview is with a maintainer of a mid-sized package registry who spent two days chasing a bad release back to a single character in a configuration file. Their name is withheld at their request, but the details are theirs.

The release that went out on a Friday#

What happened on the day of the incident?

We cut a routine release. Nothing in the changelog was scary. Two dependency bumps, a docs fix, and a change to how we parse the registry manifest. The parser change was six lines. It touched a YAML file that maps package names to their allowed version ranges. The test suite passed. The staging environment passed. We pushed to production at 16:40 on a Friday, which in hindsight was the first mistake.

When did you know something was wrong?

About forty minutes later, the error rate on install requests went from under 0.1 percent to roughly 18 percent. That number does not sound catastrophic, but it meant one in five installs was failing for users across thousands of projects. Our support queue filled within an hour. The first theory was a CDN problem. The second was a database failover. Both were wrong, and we burned four hours before anyone opened the manifest file.

The comma, not the parser#

So what was actually broken?

A single unquoted comma inside a package name field. YAML treats an unquoted comma as a list separator. Our new parser, which we had rewritten to be stricter, interpreted that line as two entries instead of one. The first entry was valid, the second was empty, and the lookup for that package started returning a null instead of a version range. Because that package was a dependency of a widely used library, the failure cascaded.

Why did the tests miss it?

Our test fixtures used package names without punctuation. Nobody had added a fixture with a comma, a colon, or a hash. The old parser was lenient; it would silently drop malformed lines and move on. The new parser was correct to reject them, but we had no test that fed it the malformed input. The bug was not in the parser. It was in our assumption that the input was clean.

How did you find it?

One of our on-call engineers dumped the parsed manifest for a failing package and compared it to the raw file. The line count did not match. That was the moment. It took about twenty minutes from there to confirm and patch. The fix was to quote the package name in the source data, but the real fix was adding a validation step that rejects malformed manifest lines before they reach the registry.

The cost and the change#

What did the outage cost?

The numbers are approximate, but we logged about 2.1 million failed install requests over 48 hours. Most resolved on retry once the patch went out, but a meaningful number of CI pipelines failed hard and had to be rerun manually. We heard from teams that lost most of a working day. That is the part that stays with you.

What changed afterward?

Three things. First, we added a fixture for every punctuation character allowed in a package name. Second, we moved the manifest validation into the build step so a bad file cannot ship. Third, we stopped doing parser changes on Fridays. That last one sounds like a joke, but it is now in our release checklist.

What is the lesson for other maintainers?

The smallest detail that mattered was not the parser logic. It was the assumption that the input was well formed. Most of the outages I have seen in open source come from that gap, where the code is right but the data is not what anyone expected. If you are rewriting a parser, write tests for the ugly inputs first. The clean ones will pass anyway.

One more thing about the fix#

Was there a moment you considered reverting?

Yes, at the four-hour mark. But reverting would have restored the old lenient parser, which had its own silent failure modes. We decided to fix forward, and that was the right call, though it cost us a night. The patch that finally went out was three lines: a quote around the package name, a regex that rejects empty list entries, and a log line that prints the raw line when validation fails. The log line is the one I am proudest of. Next time, we will see the bad line before the users do.

More articles