Flaky Test Triage

Investigate and eliminate non-deterministic behavior for: $ARGUMENTS

Purpose

Isolate tests that pass and fail intermittently without code changes, determine the underlying source of non-determinism (race conditions, order coupling, timing drift, or leaked state), and apply robust deflaking fixes.

Pre-flight

1. Confirm working tree cleanliness before running repeated test runs:

   test -z "$(git status --porcelain)" || { echo "Working tree is dirty; commit or stash changes first."; exit 1; }
   

2. Identify the target test file or test case from $ARGUMENTS.
3. Detect the test framework in use (e.g., Jest, Vitest, Pytest, Go test, Cargo test) from project configuration.

Triage & Stabilization Workflow

1. Stress Repetition to Measure Flake Frequency

Run the target test for 20 iterations without breaking early on first error so the full pass/fail ratio and flake frequency can be accurately quantified:

• Vitest:

  passes=0; fails=0
  for i in $(seq 1 20); do
    if npx vitest run <path-to-test>; then ((++passes)); else ((++fails)); fi
  done
  echo "Vitest Results: $passes passed, $fails failed out of 20 runs"
  

• Jest:

  passes=0; fails=0
  for i in $(seq 1 20); do
    if npx jest <path-to-test> --runInBand; then ((++passes)); else ((++fails)); fi
  done
  echo "Jest Results: $passes passed, $fails failed out of 20 runs"
  

• Pytest:

  passes=0; fails=0
  for i in $(seq 1 20); do
    if pytest <path-to-test> -v; then ((++passes)); else ((++fails)); fi
  done
  echo "Pytest Results: $passes passed, $fails failed out of 20 runs"
  

• Go:

  go test -count=20 -run <TestName> <package-path>
  

• Cargo:

  passes=0; fails=0
  for i in $(seq 1 20); do
    if cargo test <test_name> -- --nocapture; then ((++passes)); else ((++fails)); fi
  done
  echo "Cargo Results: $passes passed, $fails failed out of 20 runs"
  

Record the pass/fail ratio and capture stdout/stderr from failed iterations.

2. Diagnose Flake Root Cause

Categorize the failure using the common non-determinism taxonomy:

1. Timing & Asynchronous Race Conditions:

   • Symptom: Fails under high CPU load or CI runners; passes on fast local hardware.
   • Indicators: Arbitrary sleep(), setTimeout(), or unawaited promises/goroutines.
   • Root cause: Test asserts before an asynchronous background operation resolves.

2. Test Order Dependency & State Pollution:

   • Symptom: Passes when executed in isolation (--testNamePattern), fails when the entire suite runs.
   • Indicators: Leaked global variables, uncleared database records, singleton caches, or shared filesystem artifacts.
   • Verification: Use the framework's shuffle support when it is installed. For Pytest, detect optional plugins before passing their flags:

     if pytest --help 2>/dev/null | grep -q -- '--random-order'; then
       pytest <path-to-test> --random-order
     elif pytest --help 2>/dev/null | grep -q -- '--randomly-seed'; then
       pytest <path-to-test> --randomly-seed=random
     else
       echo 'No Pytest randomization plugin detected; record order-randomization as unavailable.'
       pytest <path-to-test> -v
     fi
     

     Do not pass --random-order or --randomly-seed unless the corresponding plugin advertises the option.

3. Clock & Timezone Skew:

   • Symptom: Fails around midnight UTC, during month/year rollovers, or in different local timezones.
   • Indicators: Unmocked Date.now(), new Date(), datetime.utcnow(), or reliance on implicit locale sorting.

4. Resource & Port Contention:

   • Symptom: EADDRINUSE, file locking errors, or database transaction deadlocks when run concurrently.
   • Indicators: Hardcoded network ports, static temporary file names, or shared test database schemas.

3. Apply Targeted Stabilization Patterns

Implement the appropriate pattern based on the diagnosed category:

• Replace Arbitrary Delays with Condition-Based Polling: Avoid static sleep(1000). Instead, use poll-based assertions with explicit timeouts:

  // Bad
  await sleep(500);
  expect(await getStatus()).toBe("completed");
  
  // Good
  await waitFor(async () => {
    expect(await getStatus()).toBe("completed");
  }, { timeout: 5000, interval: 50 });
  

• Enforce Clean Teardown & Mock Restoration: Ensure global mocks, timers, and database tables are cleared in afterEach hooks:

  afterEach(async () => {
    vi.restoreAllMocks();
    vi.useRealTimers();
    await cleanupDatabase();
  });
  

• Dynamic Resource Isolation: Use ephemeral ports (0 for OS-assigned port) and unique temporary folders per test run:

  const testDir = await fs.promises.mkdtemp(path.join(os.tmpdir(), "test-run-"));
  

4. Verification & Stability Certification

1. Verify stability by executing 50 consecutive iterations (stopping on any failure to observe if flake reoccurs):

   for i in $(seq 1 50); do
     <test-command> || { echo "Stability check failed on iteration $i"; exit 1; }
   done
   echo "Stability check passed: 50/50 consecutive runs passed without failure."
   

2. Run the enclosing test file with randomized test ordering when the framework supports it. Do not assume optional Pytest plugins are installed:

   # Vitest:
   npx vitest run <path-to-test> --sequence.shuffle.tests
   
   # Pytest:
   if pytest --help 2>/dev/null | grep -q -- '--random-order'; then
     pytest <path-to-test> --random-order
   elif pytest --help 2>/dev/null | grep -q -- '--randomly-seed'; then
     pytest <path-to-test> --randomly-seed=random
   else
     echo 'Pytest shuffle check unavailable: install pytest-random-order or pytest-randomly to enable it.'
     pytest <path-to-test> -v
   fi
   

3. Run the complete project test suite to guarantee zero regressions:

   npm test
   

5. Report Findings

Summarize:

• Flake Rate Before: e.g., 3 failures out of 20 runs (15% flake rate).
• Identified Root Cause: Specific race condition, state leak, or timing dependency.
• Remediation Applied: Exact code changes made to stabilize the test.
• Verification Result: 50/50 consecutive runs passed without error.

Safety Notes

• Never fix a flaky test by simply increasing an arbitrary sleep() duration; this increases test suite runtime without fixing the underlying race.
• Do not mark tests with @skip, xit, or blanket retries without addressing the fundamental root cause.
• Maintain test assertions' strictness; do not weaken expectations merely to make a test pass.