Flaky test failure

What this failure means

A test failed but the failure is not consistently reproducible. Flaky tests indicate timing dependencies, global state, or resource contention that produce different outcomes on each run.

Symptoms

Faultline looks for one or more of these log fragments:

failed on attempt
intermittent failure
test is flaky
flaky test
flakey test
timeout waiting for
race condition detected
retry attempt

Diagnosis

A test failed but the failure is not consistently reproducible. Flaky tests indicate timing dependencies, global state, or resource contention that produce different outcomes on each run.

Fix steps

1. Retry the failing test job once to confirm the failure is intermittent.

2. Reproduce reliably by running the test in a tight loop:

   # Go — rerun with gotestsum for better output:
   gotestsum --rerun-fails=3 -- -run TestName -count=20 ./...
   
   # Python:
   pytest -k test_name --count=20  # requires pytest-repeat
   
   # Jest:
   jest --testNamePattern="failing test" --forceExit
   

3. Replace fixed sleeps with deterministic waits on observable conditions, event completion, or polling with a bounded timeout.

4. Remove shared global state, reused temp directories, fixed ports, and cross-test data dependencies by using explicit setup and teardown.

5. If the test depends on time, randomness, or external services, freeze the clock, record and log the random seed, or stub the dependency so the test produces the same result on every run.

Validation

• Run the isolated test in a loop (at least 20 runs) and confirm zero failures.
• Re-run the full suite in CI and confirm no retry or flaky-flag annotations.

Why it matters

Flaky tests are most commonly caused by: shared global state between tests, async operations without explicit waits, time-dependent assertions, or reliance on external services that are sometimes unavailable.

Prevention

• Provide shared test helpers for polling, temporary resources, and fake clocks so new tests do not reintroduce timing hacks.
• Track flaky-test rates and quarantine or repair tests that exceed an agreed failure threshold.
• Run an order-randomized or repeated-test job on the main branch to catch hidden state coupling early.
• Write tests so they are self-contained, idempotent, and safe to run in parallel.

How Faultline detects it

Use faultline explain flaky-test to see the full playbook.

faultline analyze build.log
faultline explain flaky-test

Generated from playbooks/bundled/log/test/flaky-test.yaml. Do not edit directly.