Automated Test Optimization

SeaLights Test Optimization simplifies your testing process by analyzing code changes and recommending which tests are directly impacted. Here's how it works:

1. Automated Test Reporting: SeaLights automatically collects data from your tests through integrations with various testing frameworks or using its APIs.

2. Test Optimization Engine (TIA): Starting from the second build, TIA analyzes code changes and historical data to identify tests impacted by modifications.

3. Recommendations and Execution: TIA recommends skipping or prioritizing tests based on their relevance and according to a pre set Test Selection Policy. These recommendations are automatically applied and executed through integrations or by using the APIs.

Test Optimization Summary Page

App List

• List all apps with builds and testing activity within your selected timeframe.

• TIA Status: On, Partially On, Off. Partially On means only some of the test stages are executed with TIA test selection On.

• Compute Time & Savings per app. Displays total actual execution data and savings per app, aggregating test run time across all the machines.

• Potential Results with TIA ON: Shows in brackets estimated savings and optimized test compute time, in case TIA was On.

Each app is clickable to view the breakdown of the test stages.

Test Stage Breakdown

• List of all test stages which ran against at least 1 build included in the selected date range.

  • Exception! All manual test stages included in the list, even if no manual test was executed yet, in order to provide manual testers with test recommendations.

  • Test stages that are still in progress are not included in the list.

  • Unsupported testing frameworks will not receive recommendations (full test runs will occur).

• TIA Status: On, Off.

• Builds: Number of builds the test stage was running against.

• Number of Tests: Total number of tests executed across all builds. This is the accumulation of test runs, and not unique tests. In addition, average number of test executed per build.

• Compute Time & Savings: Actual execution data and savings across all builds and an average per build.

• Potential Results with TIA ON: Shows in brackets estimated savings and optimized test compute time, in case TIA was On.

Build Breakdown Page

Overview

Looking at a specific app and test stage, you can view the total number of tests and compute time and savings, as well as average per build. Graphs displaying individual build data from the last 14 days are also available.

Builds

You can switch to the Builds view to view test execution data per build.

• All tests with indication of ran / skipped and recommendation reason

• Code to test list, including all code changes in the build and their corresponding tests.