The 11-Character Filename That Broke a Build: Lessons From a CI Failure

Archi4 min read

open-sourcecishelltesting

Contents
  1. A Space in a Filename, a Three-Day Outage
  2. What Actually Broke
  3. Why It Took Three Days
  4. The Smallest Detail That Mattered
  5. The Missing Regression Test
  6. What to Check in Your Own Repository
  7. Review the Glue, Not Just the Core

A Space in a Filename, a Three-Day Outage#

The build broke on a Tuesday, and the fix landed on Friday. The cause was an 11-character filename, test data.csv, that a contributor had added to a small open source project's test fixtures. The space was legal on Linux, legal in Git, and legal in the pull request. It was not legal in the shell script that ran the test suite on every push. Nobody noticed for three days because the failure only appeared on a specific runner image and the CI dashboard had been flaky for a week.

This is a story about a tiny detail that mattered more than any architectural decision. The project had a clean module layout, a documented release process, and a maintainer who reviewed every patch. None of that helped. The bug lived in the space between two words in a filename.

What Actually Broke#

The CI job ran a shell loop that iterated over files in tests/fixtures/ and passed each one to a parser binary. The loop was written years earlier and used unquoted variable expansion. On a path with a space, the shell split the name into two arguments. The parser received a file called test and a second argument called data.csv, failed on the first, and exited with code 1.

The error message was a generic parse failure. It did not mention the filename. The maintainer saw a red X and assumed the parser had regressed. Two contributors spent most of Wednesday bisecting commits. The real cause was a single missing pair of quotes around $file in a script that had not changed in two years.

Why It Took Three Days#

The failure was real but inconsistent. Local runs on macOS passed because the contributor's shell handled the loop differently. The project's default CI runner used an older image where the parser was more strict. A second runner used a newer image where the same input produced a warning and exit code 0. The matrix had six jobs. Only two failed, and both failed intermittently because of a caching step that sometimes skipped the fixture directory.

Intermittent failures invite theories. The team considered a race condition in the parser, a flaky network mount, and a disk quota issue. Each theory cost hours. The actual answer was visible in the raw CI log, but the log was 4,000 lines long and the relevant line was buried under a wall of test output.

The Smallest Detail That Mattered#

The filename was not the problem. The problem was that the script trusted its input without quoting it. Filenames in open source repositories are user-supplied data. They can contain spaces, newlines, glob characters, and leading dashes. A shell script that expands a filename without quoting it will eventually meet one of those characters and fail.

This is not a new lesson. It is old and repeatedly taught. What made this case worth reporting is how quickly the lesson was forgotten once the build went green. The fix was a one-line change: parser "$file" instead of parser $file. It was merged with a commit message that said "quote filename in test loop." No one added a test for spaces in filenames. The project moved on.

The Missing Regression Test#

A regression test for this bug would have been trivial. Create a file with a space in its name, run the loop, assert that the parser sees exactly one argument. The project had a test for almost everything else: parser edge cases, config parsing, error codes. It had no test for the shell glue that connected those pieces.

The maintainer later said the project treated its CI scripts as infrastructure rather than code. Infrastructure gets edited in a hurry and reviewed casually. Code gets tests. That distinction is arbitrary and expensive. A shell script that decides whether a release ships deserves the same scrutiny as the parser it calls.

What to Check in Your Own Repository#

Most projects have at least one place where a filename or path is expanded without quotes. The usual suspects are shell scripts in scripts/, Makefile recipes, and CI configuration files that interpolate environment variables. A quick search for unquoted $ variables in shell files is a reasonable start. A more reliable method is to run the test suite against a directory containing a fixture with a space in its name.

Other characters to test: a leading dash (which the parser may read as a flag), a newline (rare but legal), and a glob character such as * or ?. Each one has broken a build somewhere. The space is the most common because humans type it by accident and editors do not warn.

Review the Glue, Not Just the Core#

The core code in this project was solid. The failure lived in the glue between the core and the outside world. Open source projects tend to invest review effort in the library and neglect the scripts, the Makefile, and the CI YAML. That is backwards. The glue is where untrusted input enters and where a small mistake takes down the whole pipeline.

A simple practice is to add the CI scripts to the code review rotation and require at least one reviewer who did not write the change. Another is to run the test suite in a directory whose name contains a space. That single change would have caught this bug before the pull request merged.

The project shipped a release on the following Monday, four days late. The parser was fine. The build was fine. The only thing that needed fixing was a pair of quotes around a variable that held a filename with a space in it.

More articles