Success Rate
Introducing the success rate for test cases, calculated based on their execution history, holds significant importance in the testing process. The success rate provides QA teams with a quantitative measure of the reliability and effectiveness of the test cases. The test case success rate is a crucial score for measuring test case execution performance, as a high success rate indicates a robust and dependable test case. On the other hand, a low success rate highlights potential issues or areas of improvement, signaling the need for further investigation and refinement.
Formula to Calculate Success Rate
The Success Rate of a test case is calculated based on its pass rate. This helps the QA team determine the effectiveness of the test case and facilitates test planning.
(Number of times the test case version was passed/Number of times the test case version was executed as Passed and Failed) * 100
The Status definition for Passed and Failed is defined by the Admin from the configuration screen.
Note
Required Permissions: Users should have the following QMetry Intelligence permissions to configure and generate Success Rate:
Modify Flaky and Success Rate Configuration
Calculate Flaky and Success Rate Score
Success Rate Settings
Perform the following steps for configuring Success Rate settings:
Go to Apps, select QMetry, and select QMetry Intelligence.
Select the Success Rate Settings option and select the Configuration tab.
Provide the following details for the success rate configuration:
Display Success Rate Settings
The feature provides users with the option to toggle the display of the success rate. If the feature is enabled, you can configure Success Rate computation. You can then show or hide the Success Rate column on the test case list view, test cycle > test cases tab, test execution screen, and Test Case/Acceptance Criteria section on the Jira story page. The Success Rate is also displayed on the test case detail page.
Success Rate Configuration
Test Case Executions: Mention the number of the latest test executions of a test case you want to consider while calculating the success rate. Ten test executions are selected by default.
Test Case Versions: Select the test case versions, either “Latest” or “All,” to consider while calculating the success rate. The “Latest” test case version is selected by default.
Latest: The test executions of only the latest test case versions will be considered while calculating the success rate.
All: The test executions of all test case versions will be considered while calculating the success rate.
Fix Versions: Select Fix Version(s) to consider the test executions for the test cycles with the selected Fix Version(s). “All” Fix Versions are selected by default.
Set Test Case Status.
As Passed: Select the test execution status(es) to define the test case status as “Passed”. You can select multiple statuses from the list. “Pass” status is selected by default.
As Failed: Select the test execution status(es) to define the test case status as “Failed”. You can select multiple statuses from the list. “Blocked” and “Fail” statuses are selected by default.
Click Save to save the Success Rate Settings.
To reset the settings, click Reset to Default.
To generate the success rate, click Generate.
When you hover over the Generate button, it shows the details of the user who generated the Success rate last, with the timestamp.
Once you generate the Success Rate on the Success Rate Settings screen, you can view the success rate on the test case list view, test case detail page, test cycle > test cases tab, test execution screen, and Test Case/Acceptance Criteria section in the story page.
Display of Success Rate
The following table interprets the success ratio in accordance with the display of the derived success rate calculation and color code.
Success Rate | Success Ratio |
|---|---|
| High |
| Medium |
| Low |
View Success Rate on Test Case List View
As per the Success Rate Settings done in the Configuration, the Success Rate is calculated and displayed on the test case list view.
Go to the Test Case module and make sure you make the Success Rate column visible for the list view.
Success Rate: It is calculated based on the pass rate of the test case.
You can see the Success Rate column with corresponding statistics. You can sort on the column to view test cases with higher success rates.
The info icon beside the Success Rate column displays the details of when the Success Rate was last generated and by whom.
If the Success Rate Settings are changed, the existing Success Rate will get reset.
View Success Rate on Test Case Detail Page
The generated Success Rate is displayed beside the Test Case Key at the top of the screen.
View Success Rate on Test Cycle Detail Page
Perform the following steps to view the success rate:
Go to the Test Cycle module.
Navigate to the test cycle details page and click the Test Cases tab.
Make the Success Rate column visible on the screen.
The Test Cases tab displays the Success Rate statistics.
View Success Rate on Test Execution Screen
You can view the Success Rate statistics adjacent to the test case key.
Grid View
List View
Make the Success Rate column visible on the screen.
View Success Rate in Story
The success rate of the linked test cases is displayed in the Story. It helps QA Teams to determine the stability of the story by looking at the linked test case execution performance.
Navigate to the Story, and on the Test Case/Acceptance Criteria section, make the Success Rate column visible.
You can see the Success Rate column with relevant statistics.
