Skip to content

Reproducible Test Pipelines: No Flakiness, No Chance

Flakiness points to uncontrolled inputs or dependencies. If you make time, data and isolation visible, you can narrow down the cause and make the merge gate trustworthy again.

≈ 10 min readSoftwareDelivery
$ git rev-parse HEAD
9f2c1ab

Lauf 412   FAIL  checkout.spec.ts › Rabatt gilt bis Monatsende   (2.1s)
Lauf 413   PASS  checkout.spec.ts › Rabatt gilt bis Monatsende   (2.0s)

The same commit, two runs, two results. If nothing in the code has changed and the result flips anyway, then the test reads another input that isn’t in the repository. “Just run the build again, it’s sometimes red for no reason” is therefore not a diagnosis. It is an admission that nobody knows which input that is.

A test that changes its result without a code change is reacting to another input. Often it hides in time, data or missing isolation. With integration tests, the network, cloud services or other external systems can also genuinely fluctuate. Not every dependency can be frozen; I still have to make it visible, limit it and name it in the test type. Otherwise the team gets used to its own test suite reporting green one time and red the next, and loses trust in its merge gate.

Why this is a business problem, not a test detail#

Before we get to the mechanics, here’s why the topic doesn’t end in the test directory: a flaky suite doesn’t stay a developer nuisance. It slows down the entire ability to deliver.

Martin Fowler nailed this back in 2011. A non-deterministic test is not just useless – it is a “virulent infection”. As soon as a single test occasionally turns red for no reason, everyone gets used to ignoring red builds. At exactly that moment the suite no longer protects anything, because a real failure now looks like the usual noise. The damage is not the one flaky test but the loss of trust that spreads to the whole suite.

This is not an academic argument. In 2016, Google published numbers on its Testing Blog: around 1.5 % of all test runs are flaky. That sounds like little until you scale it up. Flaky tests clog the pipeline, block branches, create duplicate bug tickets and burn developer time on a large scale – time that flows into diagnosing failures that aren’t failures at all.

And this is exactly where the loop closes to delivery performance. Accelerate (Forsgren, Humble, Kim, 2018) connects the four well-known metrics – Deployment Frequency, Lead Time for Changes, Change Failure Rate and Time to Restore – with practiced continuous integration and reliable test automation. The word “reliable” matters. An elite team that deploys on demand can only do so because the suite is a trustworthy gate. A flaky suite is not a gate but a traffic light that switches randomly – and nobody deploys based on a random traffic light. Incidentally, I deliberately always read the four metrics together: a high deploy frequency combined with a high change failure rate is worthless. Determinism in the suite is one of the conditions under which these metrics become honest at all.

So the gain doesn’t lie in “prettier tests”. It lies in shorter cycles, less friction in the team and higher delivery quality. That’s why determinism belongs in the architecture discussion and not on the QA checklist.

The three axes as a gate#

The discussion can be reduced to a simple picture. A pipeline runs from left to right: commit, build, test, merge. Above the test stage there are three dials. Only if all three are set to “controlled” does the gate turn green and the merge become possible. If one tips to “uncontrolled”, you get flaky red.

flowchart LR
  C[commit] --> B[build] --> T[test gate] --> M[merge]
  A1[TIME<br/>frozen clock] --> T
  A2[DATA<br/>seeded fixtures] --> T
  A3[ISOLATION<br/>hermetic sandbox] --> T
  T -->|all three controlled| M
  T -->|any axis uncontrolled| F[flaky red]

The value of this picture lies in diagnosis. When a build is flaky, I don’t ask “which test is acting up” but “which of the three axes is uncontrolled”. That almost always leads to the cause faster, because it structures the search. Let’s go through the axes one by one.

Axis 1: Time#

The most common hidden cause. As soon as domain code calls new Date() or Instant.now() directly, it is coupled to the wall clock – and the wall clock is guaranteed to change between two test runs. A test that checks “the discount applies until the end of the month” will behave differently on the 31st than on the 1st; a test with expiresAt in exactly 24 hours flips at time zone and DST boundaries.

The solution is not a library but a design decision: the clock becomes an injected service. Domain code never asks the operating system but always the abstraction.

interface Clock {
  now(): Date;
}

class SystemClock implements Clock {
  now(): Date {
    return new Date();
  }
}

class FixedClock implements Clock {
  constructor(private readonly instant: Date) {}
  now(): Date {
    return this.instant;
  }
}

// domain code never calls `new Date()` directly:
function isExpired(subscription: Subscription, clock: Clock): boolean {
  return subscription.expiresAt.getTime() <= clock.now().getTime();
}

In production the SystemClock runs, in the test a FixedClock(new Date("2022-01-01T00:00:00Z")). This is not an exotic idea: Java has had exactly this abstraction built in with java.time.Clock since version 8, in Ruby Timecop has done it since 2011, in JavaScript Sinon’s fake timers provide the same. I prefer the injected clock over global time freezing because it is more honest: it makes the time dependency visible in the code instead of overriding it globally.

So that it doesn’t sound too simple: time determinism is more than freezing the clock. Time zones, DST transitions, locale-dependent formatting and the iteration order of maps or sets are sources of non-determinism of their own. The frozen clock solves the big chunk, not everything.

Axis 2: Data#

The second axis is inconspicuous and does silent damage: random data. A faker.name() or an unseeded random number makes every test run unique. Most of the time that goes fine – until the one record shows up that breaks the sorting, a length limit or a Unicode edge case. Then the build is red, green on the next run, and nobody can reproduce it.

The rule is simple: deterministic data. Either fixed fixtures or a seeded random generator. The seeded route is often the better one, because it keeps the variation and still stays reproducible.

import seedrandom from "seedrandom";

function makeRng(seed = "1337") {
  return seedrandom(seed);
}

// same seed -> same sequence -> reproducible failures
function buildUser(rng: () => number): User {
  return {
    id: Math.floor(rng() * 1_000_000),
    tier: rng() > 0.5 ? "premium" : "free",
  };
}

Interestingly, exactly this rule is explicitly codified in contract testing. Pact – in use for consumer-driven contracts since 2013 – states the rule “use deterministic data, avoid random data” with no ifs or buts. The reason is revealing: if a contract contains random data, it is marked as “changed” on every run even though nothing has changed in the domain. So non-deterministic data produces not only flaky tests but flaky contracts.

Contract testing deserves a brief appreciation here, because it connects the data axis with the isolation axis. Instead of one service starting the other to test “for real”, each consumer-provider pair is checked in isolation against a shared, deterministic contract. No network, no foreign runtime, no timing – the classic sources of flakiness are ruled out by construction.

Axis 3: Isolation#

The third axis is the most fundamental and carries the fitting name hermeticity. The Bazel documentation uses “hermeticity” for builds and tests whose tools and inputs are known and controlled. This sealing off from the network, a shared database and host state removes important hidden influences; on its own, it doesn’t yet guarantee determinism if, for example, races, randomness or time remain uncontrolled in the test. A non-hermetic test can be reproducible, but it depends more strongly on its environment.

The two practical enemies of isolation are shared state and test order dependency. If test B is only green because test A wrote a row into the database beforehand, the suite is a chain that breaks with every reordering. The antidote is called shared-nothing: each test builds up its state and tears it down again.

For real dependencies such as a database or message broker, Testcontainers have become established since 2015: an ephemeral, throwaway instance per test instead of a shared, long-lived environment. The following skeleton shows the isolation discipline compactly – and it deliberately combines all three axes:

let db: EphemeralDatabase;
let clock: Clock;
let rng: () => number;

beforeEach(async () => {
  rng = makeRng("1337");                       // DATA: seeded
  clock = new FixedClock(FIXED_INSTANT);       // TIME: frozen
  db = await startEphemeralContainer();         // ISOLATION: hermetic
  await db.beginTransaction();
});

afterEach(async () => {
  await db.rollbackTransaction();               // state reset, shared-nothing
  await db.stop();
});

The transaction rollback per test is my preferred reset mechanism where it works: fast and complete. Where transactions aren’t enough, truncation between tests does the job. Only the principle matters: no test may depend on what another one left behind.

Fowler’s list of causes, by the way, maps exactly onto these three axes. He names lack of isolation, time, asynchrony or remote services, and resource leaks. Drawn as a fishbone:

flowchart LR
  ISO[lack of isolation] --> S[flaky build]
  TIME[time / clock] --> S
  ASYNC[async / remote services] --> S
  LEAK[resource leaks] --> S

Asynchrony deserves a sentence of its own, because it is the classic generator of flakiness. The most common mistake is a sleep(500) in the hope that an asynchronous operation is “probably done”. On a slow CI runner, sometimes it isn’t. The fix is never a longer wait but polling against a real condition – waiting until the state has actually been reached, with a timeout as an upper limit. “Wait until” instead of “wait how long”.

Dealing honestly with retries#

That leaves the topic everyone talks around in such discussions: the automatic retry. Almost every CI system offers to simply rerun failed tests, and if the second run is green, the build counts as green. That is tempting, because it makes the red builds disappear immediately.

The assessment is clear: retries hide non-determinism, they don’t eliminate it. As a short-term emergency patch to keep a team from grinding to a halt completely, a limited retry can be acceptable. As a permanent solution, it switches off exactly the signal you need for root cause analysis. A time-limited quarantine can be more honest: the test keeps running in a separate job and gets a named owner and a deadline. The main build doesn’t turn red because of it; the missing coverage in the merge gate, however, remains a deliberately accepted risk until the fix.

# conceptual pipeline stages
stages:
  - name: test
    fail_on_red: true          # the trustworthy merge gate
  - name: contract-verify
    run: pact can-i-deploy --to prod
  - name: quarantine
    tests: tagged("flaky")
    fail_on_red: false         # does not block merge
    on_red: open_bug_ticket    # but never silently ignored

The difference between retry and quarantine is a question of attitude. Retry says: “I hope it was chance.” Quarantine says: “I know the gap, its owner and its deadline.” Without these three pieces of information, quarantine too becomes nothing more than permanent ignoring.

Conclusion#

The common thread is simpler than the number of libraries suggests. Flakiness has a cause, but not every cause lies entirely within the test function. Freezing time, fixing data and establishing isolation clears out the most common homemade sources. External dependencies are simulated, deliberately observed or moved into a dedicated integration test whose range of fluctuation is known.

And the payoff is not “greener dashboards”. The payoff is a merge gate whose verdict the team can assess. If you dismiss flaky tests as “bad luck”, you hand your quality control over to chance. If you look for the hidden input and openly state the remaining uncertainty, you can react sensibly to a red build again.

Further reading#

Revised and translated from German with the help of AI.