"How many times do I have to run this test before I can call it stable?"
flakiness is a zero-dependency Python CLI that re-runs an arbitrary test command and answers that question with a calibrated Wilson confidence interval — and a sequential certify mode that stops as soon as the math resolves.
Re-running a test 10 times and seeing 10 passes is not evidence that it's stable. A 1 %-flaky test passes 10 in a row 90 % of the time. The existing tools (pytest-rerunfailures, pytest-flakefinder, flaky) re-run things but make you pick the run count and tell you nothing about confidence.
flakiness turns "guess and pray" into:
"I observed 0 failures in 47 trials → 95 % Wilson upper bound on the failure rate is 7.6 %. To certify ≤ 1 % you'd need ~334 more clean runs."
That last line — the "how many more runs do I need" — is the killer feature that, as far as I can tell, no other tool provides.
$ flakeguard run -n 100 -- pytest tests/test_login.py
flakeguard run — 100 runs
trials : 100
pass : 97
fail : 3
rate : 3.00%
Wilson 95% CI : [1.03%, 8.45%]
mean run duration : 0.142s
$ flakeguard certify --max-rate 0.01 --max-runs 500 -- pytest -x tests/critical.py
flakeguard certify — [OK] CERTIFIED (max-rate=1.00%, conf=95%)
trials : 381
pass : 381
fail : 0
Wilson 95% CI : [0.00%, 1.00%]
Three verdicts, three exit codes — drop straight into CI:
| Verdict | Meaning | Exit |
|---|---|---|
CERTIFIED |
Wilson upper bound ≤ --max-rate → stably ≤ threshold |
0 |
FLAKY |
Wilson lower bound > --max-rate → flakier than threshold |
1 |
INCONCLUSIVE |
Neither bound crossed within --max-runs |
2 |
git clone https://github.com/asech109/flakiness
cd flakiness
pip install -e .Runtime: Python ≥ 3.11, stdlib only. The installed binary is called flakeguard (the tool) — flakiness is just the repo name.
Don't want to install at all? python -m flakeguard ... from the repo root works too.
flakeguard run -n N [-j N] [--timeout SEC] [--target-rate P] -- <cmd>
flakeguard certify --max-rate P [-j N] [--max-runs M] [--timeout SEC] -- <cmd>
Everything after -- is your test command, run exactly as you'd run it yourself.
Vet a new pytest case before merging:
flakeguard certify --max-rate 0.01 --max-runs 500 -- pytest -x tests/test_payment.py100 runs, parallel 8, on a Go test:
flakeguard run -n 100 -j 8 -- go test -run TestParser ./parserPipe JSON into jq for CI scripting:
flakeguard run -n 200 --json -- ./run-integration my-feature \
| jq '.wilson_ci, .failure_samples[0].stderr_tail'Treat exit code 2 as success (e.g. some make targets):
flakeguard run -n 50 --success-code 0 --success-code 2 -- make smoke-test| Flag | Meaning |
|---|---|
-n N |
(run only) number of trials |
--max-rate P |
(certify only) maximum tolerable failure rate, e.g. 0.01 |
--max-runs M |
(certify only) hard cap on total runs |
--target-rate P |
(run only) report how many additional runs would certify ≤ P |
-j N |
run N invocations in parallel (you ensure test isolation) |
--timeout SEC |
per-run timeout; counted as a failure |
--success-code C |
treat exit C as pass (repeatable; defaults to 0) |
--confidence C |
confidence level for the interval (default 0.95) |
--json |
machine-readable JSON output |
--shell |
run the user command through the shell |
--quiet |
suppress live progress |
For k failures in n trials, the Wilson score interval (Wilson, 1927) for the true failure proportion is:
p̂ + z²/(2n) z · √( p̂(1-p̂)/n + z²/(4n²) )
─────────────────── ± ───────────────────────────
1 + z²/n 1 + z²/n
where p̂ = k/n and z is the standard-normal quantile for the chosen confidence. Wilson is the standard recommendation for small samples and proportions near 0 or 1, where the naive normal-approximation interval breaks down.
When k = 0, the upper bound collapses to z² / (n + z²), which gives a closed form for runs needed:
n = ⌈ z² · (1 - p) / p ⌉
For k > 0, the tool binary-searches the smallest m such that wilson_upper(k, m) ≤ p_max.
| Tool | Shift | Drift | Verdict | Runs-needed | Deps |
|---|---|---|---|---|---|
| pytest-rerunfailures | – | – | – | – | pytest plugin |
| pytest-flakefinder | – | – | – | – | pytest plugin |
| flaky (Box) | – | – | – | – | pytest/nose |
| BuildPulse, Trunk Flaky | – | – | dashboard | – | paid SaaS |
| flakiness | ✅ Wilson CI | ✅ sequential test | ✅ 3 verdicts + exit codes | ✅ closed-form / bisect | none (stdlib) |
Three modules, one concern each. ~670 lines of source + ~400 lines of tests.
┌──────────────┐
│ cli.py │ argparse, mode dispatch, formatting
└──────┬───────┘
│
▼
┌──────────────┐ ┌──────────────┐
│ runner.py │ ◀─▶ │ stats.py │
│ subprocess │ │ Wilson, │
│ pool │ │ classify │
└──────────────┘ └──────────────┘
flakeguard/stats.py— pure functions:wilson_ci,runs_needed_zero_fail,runs_needed_to_certify,classify.flakeguard/runner.py—run_once,run_manywith optionalThreadPoolExecutorparallelism + anon_resultcallback used bycertifyfor early-stop.flakeguard/cli.py— argparse, human / JSON output, exit-code mapping.
42 tests across tests/. Run them with python3 -m pytest tests/.
- Pass / fail is reduced to exit code. Sub-test granularity is the user's responsibility — run
pytest path::test_namerather than the whole suite. - No causal classification (timing? ordering? randomness?).
flakinessmeasures flakiness, it doesn't diagnose its cause. - Default parallelism = 1; raising
-jrequires test isolation (no shared temp dirs, no fixed ports, etc.). - Wilson is one defensible interval choice. Clopper-Pearson is not implemented (PRs welcome).
The original ask comes from a thread asking "What developer tool do you wish existed in 2026?" — the request was literally "Flaky Test Detector — estimate run count needed to validate new tests aren't flaky." See DISCOVERY.md for the full evidence trail; DESIGN.md for the architecture decisions; VALIDATION.md for what was tested and the two bugs found during the validation phase.
MIT. See LICENSE.