The 11-Minute CI Gap: A Q&A on the Hidden Cost of Cold Builds
Archi3 min read
The number that started the argument#
When Priya Raman joined a 40-person fintech as build engineer, she ran a simple count. The average CI job waited 11 minutes before any code compiled. Multiply by 4,200 jobs a week and the minutes became a line item nobody had budgeted for. We asked her how she found it and what changed.
Q: Eleven minutes sounds like a guess. How did you measure it?
A: Every job log has a timestamp when the runner picks up and another when the first compile step starts. I pulled 30 days of logs into a spreadsheet. Median gap was 7 minutes. The 75th percentile was 11. That tail is where the money went.
Q: What was the runner doing for those 11 minutes?
A: Most of it was image pulls. Our base image was 3.1 GB because someone had added a data science layer for a one-off notebook. Runners pulled that image fresh on every spot instance. Add package installs and a shallow clone, and you get a queue of machines downloading the same bytes.
Q: How much did this cost?
A: At 4,200 jobs weekly and a blended runner rate, the idle time alone came to roughly $9,800 a month. For a team that size, that number got attention. The build minutes were not the problem. The waiting was.
Q: What did you change first?
A: We split the image. A 600 MB build image with the compiler and nothing else. The data science layer moved to a separate workflow that runs nightly. Pull time dropped from about 6 minutes to under 90 seconds on the same network.
Q: That accounts for half the gap. What about the rest?
A: Layer caching for dependencies. We pinned the lockfile hash as a cache key, so unchanged dependencies restored from the cache instead of reinstalling. That removed another 3 to 4 minutes for most jobs. The remaining minute or two is queue scheduling, which we accepted.
Q: Did anything break?
A: Yes. Two teams had been relying on the fat image to run integration tests against a local Postgres. When we slimmed the image, those tests failed with a missing binary. We moved them to a service container and added a check that fails the build if a required tool is missing. The check now runs in 8 seconds.
Q: Your logs must have shown the gap. Why did nobody look?
A: The dashboard showed total job duration, not time to first compile. A 20-minute job looked normal because 9 minutes of it were compiling. The gap was invisible in the default view. I added a metric for pickup-to-first-step and put it on the same page as pass rate.
The caching detail that mattered#
Q: What was the most surprising part?
A: How much of the delay was unrelated to our code. The 11 minutes had almost nothing to do with the test suite or the compiler flags. It was network and disk, repeated thousands of times. Developers assume CI time is compute. Ours was mostly transfer.
Q: What would you tell a team starting this audit?
A: Pick one week of logs. Sort by pickup time, not total time. Look at the top 20 jobs by count, not by duration. The jobs that run most often are where the minutes multiply. Our lint job ran 1,100 times a week and spent 4 minutes pulling an image it did not need.
Q: Is there a number people should watch?
A: Track the ratio of idle time to work time per job. Ours was 1.4 to 1 before the fix and 0.2 to 1 after. If idle time is longer than the build itself, the runner is doing more waiting than working.
Q: Did the savings hold?
A: Six months later, the monthly runner bill was down about 55 percent, and the median time from push to green check fell from 19 minutes to 8. The surprise was not the money. It was that developers stopped batching commits to avoid the wait, which made the review queue shorter too.
Q: One thing you would undo?
A: We spent two weeks building a custom scheduler before realizing the image was the problem. The scheduler now sits unused. Measure first, then write code.
More articles