The YAML Merge Key That Silently Corrupted Our CI Pipeline for Three Weeks
Archi4 min read
Contents
The build that failed only on Tuesdays#
On a Tuesday morning, a developer pushed a one-line change to a service config. The CI pipeline that had passed for months failed at the deploy step. Rerunning the job passed. The team shrugged. Over the next three weeks, the same job failed roughly one in eight runs, always on a different commit, never reproducibly. By the time we traced it, the root cause had nothing to do with the code being deployed and everything to do with how our CI system parsed a shared YAML file.
This is a postmortem of that failure, focused on the technical chain: a merge key, a parser version mismatch, and a shared config that made the bug appear random.
The stack#
We run a monorepo with about 40 services. CI is a self-hosted runner fleet on Kubernetes, and every job pulls two YAML files: a shared ci-defaults.yml maintained by the platform team, and a per-service ci.yml that overrides a few values. Both are validated by a linter in pre-commit, and both are consumed by the CI scheduler at job dispatch time.
The shared file used YAML anchors and a merge key to reduce duplication:
yaml .base_job: &base_job timeout: 30m retries: 2 image: ci-base:2026.01
build: <<: *base_job script: make build
This is legal YAML 1.1 and widely supported. The linter passed it. The scheduler accepted it. The problem was that not every component in the pipeline was using the same YAML parser.
Two parsers, one file#
The CI scheduler is written in Go and uses gopkg.in/yaml.v3. The job runner that executes the actual steps is a Python service that uses PyYAML with safe_load. Both claim YAML 1.1 or 1.2 support. They do not treat the merge key identically in every edge case.
Specifically, when a merge key pointed to an anchor defined in a different document fragment that had been concatenated by our config loader, the Go parser resolved the merge at load time. PyYAML, depending on how the loader assembled the stream, sometimes treated the merge key as a literal string key named <<. When that happened, the runner saw a job with no timeout and no retries and fell back to defaults. The default timeout was 15 minutes. Our build sometimes took 16.
That was the whole bug: on the runs where PyYAML failed to resolve the merge, the job inherited a 15-minute timeout instead of the 30 minutes the config declared. The build would be killed mid-step. The scheduler would mark it failed. A retry would often land on a runner with a warmer cache and finish in time.
Why it looked random#
The failure rate was not random, but it looked that way from the outside. Three factors masked the pattern: , Cache state. A build with a warm module cache finished in about 12 minutes. A cold build took 16 to 18. The bug only surfaced on cold builds, which happened after dependency updates or runner restarts., Runner image version. We rolled runner images in batches. Only images built after a certain base-image bump shipped the PyYAML version with the loader behavior that mishandled the concatenated stream. About 60% of the fleet had it., Retry logic. The scheduler retried failed jobs once by default. Most failures vanished on retry, so the visible failure rate was the square of the real one.
When we finally pulled the raw job definitions from the scheduler API and compared them with what the runner logged, the mismatch was obvious. The scheduler had timeout: 30m. The runner had no timeout key at all. The merge had been dropped.
The fix and the class of bug#
We removed the merge key. The shared file now uses an explicit include that the config loader expands before either parser sees it. Both parsers get a flat document. The << key no longer appears anywhere in the pipeline. Build times are now capped at 35 minutes with a hard fail and an alert, and the runner logs the resolved config hash so we can diff what the scheduler intended against what the runner received.
This failure belongs to a family that shows up often in developer tooling: two components agree on a spec in the happy path and diverge on an edge case, and the edge case is invisible until load, timing, or scale exposes it. The YAML merge key is a common culprit because it is a non-core extension. It is not in YAML 1.2 core schema, and implementations differ on when and how they resolve it. The same class of bug shows up with environment variable interpolation, glob expansion order, and shell quoting in exec-form versus shell-form container commands.
What we changed in review#
Three process changes came out of this, all cheap:
- Pin parser versions in CI images and assert them at job start. The runner now logs its PyYAML version and the scheduler logs its Go YAML version on every run. A mismatch across the fleet surfaces in a dashboard instead of in a flaky build.
- Ban merge keys and anchors in pipeline config. Anchors are fine in human-maintained docs. In machine-consumed config, explicit expansion is easier to debug and easier to validate.
- Alert on default-value fallbacks. Any time the runner applies a default for a field that the scheduler explicitly set, that is now a warning, not a silent substitution. The timeout case would have paged us on day one.
The bug cost roughly 40 engineer-hours of investigation across three weeks and one missed release. The fix took an afternoon. The expensive part was not the change. It was the three weeks we spent assuming the failure was in the code we were deploying, when the code was fine and the config parser was not.
More articles