Why CI Cache Misses: A Technical Checklist for Build Locality
Archi4 min read
Contents
Why cache misses happen#
A CI cache miss usually looks like a performance problem. The technical root cause is often a key collision or a key mismatch. The cache system has the artifact, but the computed lookup key does not match the stored key. When that happens, every job restores nothing and rebuilds from source.
Understanding the exact string used for the key is the first step. Most cache APIs accept a string and return a byte-exact artifact. If the string changes by one character, the miss is intentional. If it does not change when input changes, the miss is hidden because you restore a stale artifact.
Check the key inputs#
- Use the full lockfile path instead of the package manifest alone. An npm project has package.json and package-lock.json. The lockfile is the deterministic input. Package manifests change often, lockfiles change only when resolved dependency versions change. Keying only on package.json causes restores of old dependency trees after a lockfile update.
- Hash the content, not the modification time. Some CI examples use hashFiles on a directory tree. If the tool includes mtime, every checkout changes the hash because Git resets file timestamps to the current time. Use git status flags or a content hash. GitHub Actions
hashFilesis content-based, but custom shell pipelines often fall back todate. - Include the build toolchain version. A C++ project built with GCC 12 and cached with GCC 13 will produce incompatible object files. Put the compiler version in the key. The same applies to Python versions. A wheel built for Python 3.11 can fail on a Python 3.12 runner.
- Include only the files that affect output. Some monorepo caches key on the entire repository. That invalidates every package when a README changes. Use tool-specific file scoping, such as
./packages/api/**/*.lockor a hashed set of project manifests.
Watch the restore policy#
A cache miss can also come from what happens after the first lookup. Most systems support restore keys that allow partial matches. If you define a restore key, the job may pull a previous cache and then treat the build as warm. That is a correctness failure as well as a performance one. A dependency cache should be restored only when the key exactly matches the current dependency set.
Set the restore policy to exact match for compiled artifacts. For source dependencies that are slower to download than rebuild, a partial match can be acceptable. Separate those cases in configuration. Do not mix a fallback restore key for binaries with one for source tarballs.
Check absolute path drift#
Compiled cache artifacts can miss for reasons unrelated to the cache key. Many languages embed absolute build paths. If a job runs under /home/runner/work/repo and the restore job expects /build/project, the restored object files carry the old path. The linker or compiler may reject them if it checks dependency files. Use relative path settings, such as Rust's --remap-path-prefix, Go's -trimpath, or Bazel's workspace-relative paths.
The same problem appears with Docker layer caching. A RUN command that copies source into the image records the path. Running the build on a different container version or different WORKDIR changes the layer context. Keep the WORKDIR constant across cache restore and normal runs.
Inspect the actual key#
When a miss still happens, print the key value. Add a step that outputs the resolved cache key after all inputs are computed. Compare it between the failing job and a known-warm job. The difference will show which input is not deterministic.
Common sources of non-determinism:, Build IDs injected by the runner, Random temporary directory names passed to the key, Environment variables that contain token values, Git commit timestamps included in the key, Newlines added by shell scripts when building the key string
Remove each one. A cache key should be a pure function of source content and toolchain versions. Any other input destroys locality.
Partition remote caches#
Shared caches add a second layer: remote cache coordination. Tools like Bazel and Turborepo can upload cache entries to a central store. A miss then means the local job did not find an entry that another job uploaded. The technical root cause is often a mismatch in the action hash.
Bazel computes action hashes from command line flags, environment variables, and toolchain declarations. If one developer runs bazel build //... and another runs bazel build --compilation_mode=opt //..., the action hashes differ. Keep the set of build flags identical across all jobs and local developers by using a checked-in .bazelrc or a shared toolchain file.
Turborepo and Nx use task hashes from package dependency graphs. If the monorepo task definition differs between machines, the hash differs. For example, Turborepo includes the outputs array in the task hash. A missing outputs entry turns a previously cached task into a miss. Check the task definition with turbo run build --dry-run and compare the hash fields.
List the cache directories#
Finally, verify that the directories you cache are the directories the build reads. A pip cache is separate from the installed site-packages. Caching ~/.cache/pip speeds up pip install only when the package installer still hits the network and reassembles packages. If the project uses a vendored wheel directory, cache that directory instead. The same confusion appears with Docker buildx cache, which is separate from the image layer cache. Buildx cache must be configured with --cache-from and --cache-to, or the restore does nothing.
Run a build with all caches disabled and watch the path list. Then enable the cache and compare the paths. The read path must match the write path. A one-character difference in a directory name is the simplest miss to fix and the easiest one to overlook.
More articles