Automation Health
2 min read
Automation Health answers two questions about your pipelines: what are they running that Hawzu cannot see?, and where is their time going?
Both are about the gap between the suite your CI actually executes and the test cases your project knows about. A pipeline reporting two hundred tests that map to forty test cases is not covering forty things well — it is telling you about a hundred and sixty you have no record of.
Opening the report
Section titled “Opening the report”Open the Reports area for your project and select the Automation Health card — “What your pipelines run that Hawzu cannot see, and where their time goes.”
The report has a single filter: a window of Last 7, 14, 30, or 90 days. Everything below is measured over that window.
The three metrics
Section titled “The three metrics”| Metric | Meaning |
|---|---|
| Unmatched tests | Reported by a build, matching no test case |
| Seen repeatedly | Unmatched in more than one build |
| Timed tests | With a duration recorded |
Seen repeatedly is the one to act on. A test unmatched in a single build might be a one-off experiment; a test unmatched in every build for a month is a permanent hole in your traceability.
Automation Hawzu cannot see
Section titled “Automation Hawzu cannot see”Every test a build reported that matched no test case, with Test, Builds, Last result, and Last seen.
As the report puts it: “These ran in a pipeline and matched no test case. Add the test case code to the test’s name, or create the case it covers.”
Two ways to close a row:
- The test covers something you already have a test case for. Put that test case’s code in the test’s name, or state it explicitly in the report. See How Results Map to Test Cases.
- The test covers something nobody wrote a case for. Create the test case, then add its code to the test.
When the list is empty it says so: “Every test a build reported mapped to a test case.”
Where the time goes
Section titled “Where the time goes”Timing for automated tests across builds, with Test, Runs, Average, Slowest, and Spread.
Spread is the slowest run divided by the average, and it is the interesting column. A test that always takes nine seconds is predictable. A test averaging nine seconds with a slowest of forty is not — it is a flake or a timeout waiting to happen, and Hawzu flags a spread of 3× or more in orange.
The report’s own summary: “Averaged across builds. A large gap between average and slowest is a test that is about to start timing out.”