Why Your Test Suite Passes Alone and Fails Together: The Shared State Problem

Archi4 min read

testingdebuggingci

Contents
  1. The symptom every developer recognizes
  2. Module scope is the usual suspect
  3. Order dependence is the telltale
  4. The filesystem and the database
  5. Environment variables and the process
  6. Parallelism makes it worse and easier to find
  7. A quick way to bisect the leak
  8. What clean isolation actually requires
  9. The honest limit

The symptom every developer recognizes#

You run a single test file. Green. You run the whole suite. Red. You rerun just the failed test in isolation and it passes again. Nothing in the code changed between runs, so the difference has to be something the tests do to each other. That something is shared state, and in most test runners the sharing happens through four channels: module-level variables, the filesystem, the network (or a local database), and the process environment. Each channel leaks differently, and each one leaves a different fingerprint.

Module scope is the usual suspect#

Most languages load a module once per process. Anything at the top level of that module, a cache dictionary, a counter, a connection pool, a let binding, survives across every test in the file and often across the whole run. Test A populates the cache and asserts one path. Test B assumes an empty cache and asserts another. Run B alone and it passes, because nothing filled the cache first.

The root cause is that module initialization is treated as a constant when it is actually mutable state. The fix is not to sprinkle resets everywhere. Find the state that gets written during a test and ask whether it should be per test. Dependency injection, constructor arguments, or a factory that returns a fresh object per test removes the leak at the source.

Order dependence is the telltale#

If a suite only fails when tests run in a particular order, you have proven the dependency exists. Most runners offer a random-order or shuffle mode. Turn it on in CI for a week and collect the failures. A test that fails under shuffle but passes under fixed order is not flaky in the mystical sense. It has a hidden input, and the hidden input is whatever another test left behind.

The filesystem and the database#

A test writes output.csv to the current directory. Another test reads output.csv expecting its own fixture. Both pass alone. Both fail together, or worse, one passes with the wrong data and the assertion is simply wrong in a way nobody notices for months.

The same pattern shows up with databases. A test that inserts a row and never deletes it will make a later count assertion fail. Transaction rollback per test, a fresh schema per run, or a unique schema name per worker all solve it. The key detail is isolation at the level the test actually touches, not a comment saying tests should clean up after themselves.

Environment variables and the process#

Setting process.env.API_URL in one test and not restoring it is a classic. So is changing the working directory, monkeypatching a global, or seeding a random number generator with a fixed value. These leak because the process is the shared resource. On CI they often hide, because each job runs one file or one shard, and nobody runs the full suite locally with the same parallelism.

Parallelism makes it worse and easier to find#

Test runners that shard across workers expose shared state fast. Two workers hitting the same database table, the same temp filename, or the same fixed port will collide intermittently. That intermittency is not random. It is a race with a repeatable structure: who gets there first varies, so the failure varies.

Start by giving each worker its own namespace: a database schema suffix, a temp directory from the OS, a port assigned by the kernel (bind to port 0). Then look for tests that hard-code a name like test-db or localhost:5432.

A quick way to bisect the leak#

Run the suite with one test excluded at a time, or split it in half and run each half with the failing test appended. You are looking for the smallest pair of tests where one pollutes and one depends. With two candidate files, run them in both orders. If the failure follows the order, you have isolated it to a handful of lines. This takes minutes and beats reading the whole suite.

What clean isolation actually requires#

The principle is simple: every test should see the world as if it were the first test to run. In practice that means four habits, applied where the language allows. , Fresh objects for anything the test mutates, created inside the test or in a setup hook that runs per test, not per file., A per-test or per-worker data store, never a shared fixed name., Environment changes made through a helper that restores the previous value on teardown, including on failure., No reliance on wall-clock time, random seeds, or network reachability unless the test creates and controls them.

One more thing that saves time: failure messages should say what state was expected. A test that fails because a cache was not empty should report the cache contents, not just expected true, got false.

The honest limit#

Some state is genuinely global: a singleton HTTP client, a hardware device, a browser instance. You cannot inject your way out of all of it. What you can do is make the shared resource explicit and serialized, so the dependency is visible in the test rather than hidden in the runner. That is the difference between a suite that fails mysteriously once a month and one that fails with a message you can act on. The bug is never the test runner. It is almost always a variable that outlived the test that set it.

More articles