Savings Calculation

Test Optimization is a core SeaLights functionality designed to accelerate development velocity by identifying and skipping tests that are not impacted by recent code changes. By analyzing the delta between builds, SeaLights intelligently recommends a subset of the test suite that is relevant to the specific code modified, ensuring quality while drastically reducing execution overhead.

The "savings" metrics within SeaLights provide a transparent view of the time and resources conserved by transitioning from traditional, exhaustive testing to an optimized, impact-based approach.

Core Metrics Definitions

To provide a precise view of optimization efficiency, SeaLights utilizes the following metrics:

• Original Time: The total compute time it would have taken to execute the entire relevant test suite if no optimization was applied.
• Actual Time: The total compute time actually spent executing the tests that ran in the optimized cycle.
• Saved Time: The total compute time conserved. This specifically accounts for the time of tests recommended for skipping by SeaLights and that were actually skipped.
• Original Test Count: The total number of tests in the suite before optimization.
• Actual Test Count: The number of tests that were actually executed in the optimized run.
• Saved Test Count: The number of tests recommended for skipping by SeaLights that were not executed.

Note on "Compute Time" Compute time refers to the total machine time required to run tests, not the elapsed "wall-clock" time. In parallel environments, if 10 machines run for 1 minute each, the compute time is 10 minutes, even if the user only waited 1 minute.

Calculating Optimization Savings

1. Estimating "Original" Values

To determine what the run would have looked like without optimization:

• For tests that ran: SeaLights uses their real execution duration.
• For tests skipped by Test Optimization: SeaLights estimates their duration by calculating the average of their last five historical runs.
• Only tests reported in the current execution context are included, ensuring deleted or obsolete tests do not skew the data.

2. The Savings Formula

The percentage of savings is the primary KPI for the efficiency of your optimization policy.

Time Savings Percentage: The formula calculates the ratio of skipped time relative to the total potential time:

$$\text{\% Savings (Time)} = 1 - \left( \frac{\text{Actual Time}}{\text{Actual Time} + \text{Skipped by SeaLights Time}} \right)$$

Count Savings Percentage: The same logic applies to the volume of tests:

$$\text{\% Savings (Count)} = 1 - \left( \frac{\text{Actual Test Count}}{\text{Actual Test Count} + \text{Skipped by SeaLights Count}} \right)$$

3. Defining Saved Time

SeaLights is strict about what constitutes "Savings." Saved Time only accounts for the time of tests that met two criteria:

1. They were officially recommended to be skipped by the SeaLights analysis.
2. They were actually skipped (not executed) in the build.

---

FAQ

Why doesn't "Saved Time" always equal "Original Time - Actual Time"?

"Saved Time" is specifically tied to SeaLights' recommendations. If a test was skipped for other reasons (e.g., manually disabled in code or failed due to environment issues before the test started), it is not credited to SeaLights Test Optimization savings.

How does SeaLights handle new tests?

New tests have no historical data. SeaLights typically defaults to running new tests to establish a baseline and ensure coverage, meaning they will contribute to "Actual Time" and "Actual Test Count" in their first run.

What happens if a test's execution time fluctuates wildly?

SeaLights uses a rolling average of the last five runs to smooth out environmental noise and provide the most accurate "Original Time" estimate possible.