Ctrlk
Contact SupportProduct
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Test Optimization Strategy

The Test Optimization Strategy page is the centralized governance hub for managing how SeaLights optimizes your test suites. It provides a single entry-point to configure, monitor, and manage optimization levels across your entire organization.

The interface supports a consolidated view that distinguishes between Cross-App test stages and App-Level test stages.

Read more about the differences between the two types on Test Optimization Models: Cross-App vs. App-Level.

Optimization Strategy Levels

In the Cross-App view, users can track the current strategy level of each stage, which dictates the aggressiveness of the Test Optimization recommendation algorithm:

Level
Behavior

Conservative

Basic Test Optimization. Runs every test identified as impacted. No additional pruning. Ideal for CD/frequent deployments.

Moderate

Default. Uses the basic Test Optimization algorithm plus a 80% "Maximum Capacity" filter. Prioritizes high-confidence testing for standard CI cycles.

Aggressive

Uses the basic Test Optimization algorithm plus a 60% "Maximum Capacity" filter. Selects the most critical subset for the fastest possible feedback.

In the App-Level view, there is not indication for strategy levels, as it always operates at the Conservative level (basic Test Optimization).

Read more about the recommendation algorithm and strategy levels impacts on Generating Test Recommendations.

Strategy Views (Consolidated Entry-Point)

The page features a primary toggle to switch between different strategy scopes.

  • Cross-App (Default): Displays "Unbounded" test stages, those that apply across multiple services or the entire organization.

  • App-Level: A flat list of all "Bounded" test stages (stages tied to specific applications). Unlike previous versions (old UI: Savings Summary), these are no longer grouped by application, allowing for superior filtering and sorting across the entire portfolio.

  • Integration Build (Conditional): This option appears only for customers using Integration Builds. It displays a table identical to the App-Level view but filtered exclusively for integration build applications.

The Strategy Table

The table serves as the primary management interface.

Default Columns:

  • App (App-Level & Integration Build views only): Displays the specific application name associated with the bounded test stage. This column is not present in the Cross-App view as those stages are unbounded.

  • Test Stage: The name of the stage (e.g., System-Integration-Tests). When applicable, the project-ID is chained to the test stage. Read more about test-project-ID.

  • Status: A master toggle indicator (On/Off). When Off, optimization is disabled, and the table displays a dash (-) for all metrics.

  • Optimization Level (Cross-App only): Displays the current optimization strategy setting (Conservative, Moderate, or Aggressive).

  • 30-Day Savings: Your primary ROI metric, showing the percentage of time saved out of the full test suite over the last 30 days.

Customizable/Hidden Columns (App-Level & Integration Views):

In the App-Level and Integration Build views, users can click the Customize button to toggle the visibility of additional data columns:

  • Builds: Total number of builds optimized in the last 30 days.

  • Total Executed: Total count of tests run across all optimized builds.

  • Avg. Executed / Avg. Savings: Average counts per build.

  • Compute Time: Total actual time spent executing the reduced test sets.

  • Avg. Compute Time: Average execution time per cycle.

Breakdown & Configuration

Breakdown and configuration paths differ depending on the current view:

App-Level & Integration Build Views

Hovering over a row reveals the buttons:

  • Breakdown Charts: Opens a side panel showing historical trends and savings distributions.

  • Breakdown Analysis & Settings: (Admin/DevOps only) Opens the Savings Breakdown page in a new tab with pre-filtered data for that specific stage. Configuration of Test Optimization is managed on this page.

Cross-App View

Clicking a row navigates you to the Specific Stage Strategy page, that includes:

  • Configuration: Fine-tune optimization strategy rules (by Admin/DevOps only).

  • History: Access the full Breakdown Analysis and the Breakdown Charts for any previous month.

Read Enabling & Configuring Test Optimization, for more details on configuration.

FAQ

Why do some rows show a dash (-) in the savings column?

This happens if the status for that stage is set to Off. SeaLights only calculates savings data for stages where Test Optimization is actively enabled.

What happens if I don't report test results back to SeaLights?

If results are not reported, the system cannot "clear" the impacted tests. They will continue to be recommended in the next cycle as "impacted," which will decrease your optimization savings.

Does Test Optimization respect my "Ignored Code" settings?

No. To ensure the highest product quality, Test Optimization provides recommendations for your entire codebase, including parts you may have set to "ignore" in other areas of the SeaLights platform.

Can I still see the old "Savings Summary" page?

For a transitional period, existing customers who have not yet migrated to the "New Experience" will still have access to the legacy Savings Summary view. New features and the consolidated entry-point are exclusive to the Test Optimization Strategy page.

Was this helpful?