Flaky test hunter: rerun tests N times and find the unstable ones

What it does

Finds flaky tests from recorded evidence instead of guesses.

• Runs the given test command N times and saves each run's exit code, duration and output (log files)
• Aggregates per-test results from pytest, jest, vitest, mocha, go test, unittest, cargo test and TAP output
• Reports tests that both passed and failed as flaky, and tests that failed every time as always failing, with failing run numbers and failure rates
• Shows min, median and max run time and records timed-out runs
• Includes a reference of common causes (order dependence, time, randomness, network, async, parallelism) and how to confirm them

How to use

python scripts/flaky_hunter.py --runs 20 -- pytest -rA -q tests/test_api.py
python scripts/flaky_hunter.py --runs 20 --timeout 120 --shell "npx jest --verbose --runInBand"

Results go to flaky-report/report.md, results.json and logs/. The agent reads the failing logs and the test code, then reports likely causes and fixes.

Requirements

• Python 3 (standard library only)
• An environment that can run the project's tests

Caveats

• The command is executed N times as given; run suites that touch databases or external services in a test environment.
• Per-test results need a reporter that prints one line per test (-rA, --verbose, -v). Unrecognized output still gives run-level results and logs.
• No failure in N runs does not prove a test is not flaky.