Flaky Tests
2 min read
The Flaky Tests report highlights tests that flip statuses across runs — bouncing between pass and fail rather than failing consistently. It gives you a stable triage shortlist so you can fix the noisiest tests first and tell genuine bugs apart from flakes.
Filtering
Section titled “Filtering”- Search by title or code: Find a specific test case.
- Min flips: Set a minimum number of pass↔fail transitions a test must have before it appears.
- Classification: Filter to Flaky or Unstable tests.
- Folder: Limit results to a folder.
Key metrics
Section titled “Key metrics”The report summarizes the tests with run data in scope:
- Tests with data
- Flaky
- Unstable
- Stable
Probable breakages and folder instability
Section titled “Probable breakages and folder instability”- Probable breakages: Tests whose recent signals point to a real break rather than a flake — for example, a failing streak or open defects.
- Folder instability: Each folder’s instability percentage, calculated as (Flaky + Unstable) ÷ tests with data, so you can see which areas misbehave the most.
Reading the flaky tests table
Section titled “Reading the flaky tests table”The main table lists each flaky test with:
- Streak: How many consecutive recent runs ended with the same status (for example, “3× Failed”).
- Last failed: The most recent run that ended Failed or Blocked.
- Flips: The number of pass-to-fail or fail-to-pass transitions across the recent runs in scope. Higher means more bouncing.
- Fail %: Failed runs ÷ total runs in the flakiness window.
- Open defects: Defects linked to the test that are not closed — a value greater than zero often signals a real bug rather than a flake.
- Classification: Whether the test is classified as Flaky or Unstable by the flakiness model.
Recent-run sparklines
Section titled “Recent-run sparklines”Each row carries a short strip of dots — one dot per run, oldest to newest — so a pattern is visible at a glance.
Every run counts as its own dot, including individual CI builds. A test run automated against a pipeline that built forty times contributes forty dots, not one. Without that, a suite that runs nightly would look far more stable than it is.
CI builds are drawn with a ring around the dot rather than a different colour, so status stays readable and the source is still distinguishable. Hovering a dot names both: Failed • CI build • 12 Mar 2026, or Passed • Manual run • 12 Mar 2026.
Exporting
Section titled “Exporting”Download the report as a PDF or as Excel (XLSX). The PDF dialog lets you choose sections — cover page, executive summary, key metrics, probable breakages, folder instability, the flaky tests table, and a legend explaining flips, fail rate, streaks, and the classifications.