Execution Analysis & Insights
Execution Analysis provides a real-time execution cockpit for test runs inside a release, along with post-execution insights for reporting and review.
It is designed to help teams answer one core question:
“Based on what is happening right now (or what already happened), what should we do next?”
Execution Analysis is not just a report.
It is a decision-support view that combines outcomes, progress, quality health, productivity, ownership, and risk signals into a single place.
When to Use Execution Analysis
Execution Analysis is useful in two phases, with clear emphasis on live usage.
During Execution (Primary Use)
Use Execution Analysis to:
- Monitor execution progress in real time
- Identify failures, blocks, and execution gaps early
- Detect risk signals before completion
- Balance workload across users
- Decide whether to continue, pause, or refocus testing
After Execution (Secondary Use)
Use Execution Analysis to:
- Review execution quality and outcomes
- Analyze team productivity
- Identify bottlenecks and improvement areas
- Generate reports for stakeholders
- Archive execution insights for future reference
Execution Overview Metrics
The top section displays raw execution outcomes and progress indicators.
What this shows
- Total Testcases: Total number of test cases in the execution
- Executed: Number of test cases that have been executed
- Coverage: Percentage of execution coverage achieved
- Pass Rate: Percentage of passed test cases
- Failure Rate: Percentage of failed test cases
- Block Rate: Percentage of blocked test cases
These metrics represent facts, not interpretations.
A high pass rate does not automatically mean low risk.
Coverage, failures, and blocked tests must be considered together.
Execution Status Breakdown
This section shows how test cases are currently distributed by execution status:
- Passed
- Failed
- Blocked
- Skipped
- Not Executed
Why this matters
- A large number of blocked tests can hide real quality issues
- Skipped tests reduce effective coverage
- Not Executed tests indicate incomplete execution
This breakdown helps teams quickly understand execution completeness and stability.
Health & Productivity Scores
This section introduces opinionated but transparent indicators to help teams assess execution quality and efficiency.
Health Score
The Health Score represents overall execution quality confidence.
It is derived from:
- Pass rate
- Failure rate
- Execution coverage
Purpose:
“Is this execution safe to move forward with?”
A high Health Score indicates strong confidence in execution quality, not just completion.
Productivity Score
The Productivity Score reflects execution efficiency.
It is influenced by:
- Test execution velocity
- Average tests executed per user
- Execution time efficiency
Purpose:
“Is the team executing efficiently and sustainably?”
Health and Productivity are intentionally separate:
- A team can be productive but risky
- A team can be healthy but slow
Testcase Source Analysis
This section shows where execution scope comes from.
Testcases can originate from:
- Requirements
- Test Suites
- Manual Selection
Why this matters
- Requirement-heavy executions validate traceability
- Suite-heavy executions validate regression strategy
- Manual-heavy executions may indicate ad-hoc testing or gaps in suites
Understanding testcase sources helps teams:
- Validate execution planning
- Identify over-reliance on manual selection
- Improve test suite quality over time
Execution Velocity & Forecast
This section focuses on execution momentum.
What it shows
- Average Tests per User
- Execution Velocity
- Completion Forecast
- Remaining Tests
- Resource Utilization
How to use this
- Predict whether execution will finish on time
- Identify underutilized or overloaded resources
- Adjust assignments dynamically during execution
Velocity is execution-specific, not a performance score for individuals.
User Performance Analysis
This section provides execution-level workload visibility, not individual evaluation.
What it shows
- Assigned vs unassigned tests
- Tests executed per user
- Pass rate per user
- Execution efficiency indicators
Important Notes
- This is not a leaderboard
- It is meant for load balancing and planning
- A high number of unassigned tests is a system-level risk
Use this view to:
- Distribute work evenly
- Avoid execution bottlenecks
- Improve team coordination
Risk Assessment
Execution Analysis continuously evaluates execution risk based on live data.
Risk Signals Considered
- Failure rate
- Block rate
- Execution coverage
Risk Levels
- Low Risk
- Medium Risk
- High Risk
A High Risk execution does not block progress automatically, but signals that attention is required.
Bottlenecks & Recommendations
This section translates risk signals into actionable guidance.
What it provides
- Identified issues (e.g., high failure or block rate)
- Priority indicators (Urgent, Important, Medium)
- Clear next-step recommendations
Examples
- Investigate failed test cases exceeding threshold
- Resolve blocked tests impacting coverage
- Assign unassigned tests to improve velocity
This system is advisory, not enforcement-based.
Best Practices
During Execution
- Monitor Health and Risk indicators continuously
- Address blocked tests early
- Balance workloads using user insights
- Use velocity signals to adjust scope if needed
After Execution
- Review failures and recommendations
- Analyze productivity trends
- Improve test suites and planning
- Archive reports for future reference
Summary
Execution Analysis is designed to help teams:
- Act early
- Reduce risk
- Improve execution quality
- Make informed decisions
Think of it as:
A live execution cockpit first,
and a post-execution report second.
Next Steps
- Learn about Creating Executions
- Explore Release Management
- Review Release Reports & Export