Full Run Policy
Last updated
Last updated
Test Impact Analysis (TIA) provides the option to recommend a full test run based on policy. This strategic approach offers two key benefits:
Enhanced Mapping Accuracy: Statistical modeling forms the foundation of TIA, and the more test executions TIA captures, the more precise its connection map becomes. Therefore, scheduling regular full test runs ensures that TIA maintains accurate mappings, addressing changes in tests and modifications to test configurations, identifying deleted or renamed tests, and improving the statistical modeling of relatively new tests.
Comprehensive Coverage for Critical Changes: When dealing with critical or sensitive code modifications, a full test run is essential to ensure thorough coverage and comprehensive testing. This can be achieved by either setting up a recurring full run or utilizing the option to designate the next build as the full run.
While periodic full test runs are crucial for TIA's accuracy, it's essential to balance this necessity with the goal of maximizing savings and adhering to the TIA's purpose of reducing test execution time. Here's a recommended frequency for full runs based on test stage execution patterns:
For test stages running a few times a day: Set the full run policy to once a day, typically at the start of the day (00:00).
For test stages running a few times a week: Set the full run policy to once a week, typically on Monday at the start of the week (00:00).
For extensive manual test stages: Set the full run policy to once a month, typically at the 1st of the month (00:00).
For any other use cases: Set the full run policy to every X builds for a specific test stage.
Important! After utilizing TIA for a while, evaluate the savings achieved for a particular app and test stage. If savings are not as expected, consider decreasing the frequency of full runs to optimize testing efficiency.
Define a default policy: Apply to all apps and test stages with TIA enabled via the SeaLights Settings / TIA page.
Customize for specific needs: Tailor the policy for individual apps, branches, and test stages through the Settings page or the Test Optimization - Build Breakdown page (using the TIA configuration button).
By leveraging both pre-defined and customizable policies, you can ensure SeaLights Test Optimization delivers recommendations that perfectly align with your unique testing strategy.
