Generating Test Recommendations

After identifying all code changes in a new build and leveraging the Code-To-Test mapping, TIA determines which tests are affected by these changes. It subsequently compiles a list of recommended tests, which is accessible for viewing or consuming via an API and can be integrated back into the test framework for executing only the necessary tests. Alternatively, the list of tests can be incorporated into the customer's CI pipeline.

By utilizing TIA, teams can run only the relevant tests associated with code changes, saving time and enhancing efficiency.

Test Recommendation Principles

TIA follows specific principles to construct a precise list of essential tests that ensures high quality and instills confidence in teams and organizations. These principles guide the creation of the test recommendation list:

• Tests impacted by code changes. Meaning tests related to modified methods.

• Tests that failed in the last test run on the specific test stage for comprehensive coverage.

• Pinned tests, which must consistently run.

• New tests not yet mapped to any code section.

• Dependent tests, which should run alongside related recommended tests.

• Previously recommended tests intended for the previous test recommendation but not executed, often due to failures or blockers.

Tests that do not fall into any of the principles above are excluded from the recommendation list and automatically skipped. This refined test list maintains quality while reducing the test suite.

Test Selection Policies

SeaLights empowers you to customize the recommended test list to perfectly align with your specific testing needs. This flexibility allows you to:

• Broaden coverage: Ensure comprehensive testing by incorporating additional tests.

• Optimize test volume: Prioritize critical tests to streamline execution time.

SeaLights empowers you to find the perfect balance between thoroughness and efficiency in your testing process, ultimately saving time and resources.

Test Selection Policies

The Impact On Code Coverage

TIA offers the advantage of skipping irrelevant tests for each build and release. Consequently, this selective approach may result in lower overall code coverage. This can be a concern if quality gates within an organization rely on code coverage metrics and require a specific threshold for validation.

Similarly, other systems that depend on full test execution, such as SonarCube, may be affected by TIA's selective test runs. It's essential to recognize this potential impact and communicate it proactively with all relevant stakeholders.

In cases where all tests must run for specific reasons, perhaps due to a critical and sensitive code change necessitating a comprehensive system test, you have the option to temporarily disable TIA, or to choose a full run policy for the next build to ensure all tests are executed without skipping any. This allows you to meet the requirements of sensitive code changes while maintaining comprehensive coverage.

Remember that balancing the advantages of TIA's efficiency with the need for comprehensive testing is crucial. Effective communication with stakeholders is key to managing expectations and ensuring quality throughout the development process.

TIA - How our agents work?

As part of TIA, one of the most important capabilities is the ability of the agents to skip tests that shouldn't be run. In some technologies, this capability is achieved by extending the test runner to query Sealights for a list of tests that should be excluded from the run, skipping them while allowing the relevant and new tests to run. In other technologies such as JUnit and TestNG Sealights does that for you.

Skipping tests for supported technologies is being done by modifying the behavior of the test runner at runtime - our agents instrument the code and make calls to Sealights APIs to get the list of tests that should be skipped and skip them for you.

As part of the design of the system, the agents are designed in a way that if they fail to retrieve the information from the server (for any reason whatsoever), the agent will revert to full run, meaning, run all the tests.