# Savings Calculation

**Test Optimization** is a core SeaLights functionality designed to accelerate development velocity by identifying and skipping tests that are not impacted by recent code changes. By analyzing the delta between builds, SeaLights intelligently recommends a subset of the test suite that is relevant to the specific code modified, ensuring quality while drastically reducing execution overhead.

The "savings" metrics within SeaLights provide a transparent view of the time and resources conserved by transitioning from traditional, exhaustive testing to an optimized, impact-based approach.

### Core Metrics Definitions

To provide a precise view of optimization efficiency, SeaLights utilizes the following metrics:

* **Original Time:** The total compute time it would have taken to execute the entire relevant test suite if no optimization was applied.
* **Actual Time:** The total compute time actually spent executing the tests that ran in the optimized cycle.
* **Saved Time:** The total compute time conserved. This specifically accounts for the time of tests recommended for skipping by SeaLights and that were actually skipped.
* **Original Test Count:** The total number of tests in the suite before optimization.
* **Actual Test Count:** The number of tests that were actually executed in the optimized run.
* **Saved Test Count:** The number of tests recommended for skipping by SeaLights that were not executed.

{% hint style="info" %}
**Note on "Compute Time"** \
Compute time refers to the total machine time required to run tests, not the elapsed "wall-clock" time. In parallel environments, if 10 machines run for 1 minute each, the compute time is 10 minutes, even if the user only waited 1 minute.
{% endhint %}

### Calculating Optimization Savings

#### **1. Estimating "Original" Values**

To determine what the run *would* have looked like without optimization:

* For tests that **ran**: SeaLights uses their real execution duration.
* For tests **skipped by Test Optimization**: SeaLights estimates their duration by calculating the average of their last five historical runs.
* Only tests reported in the current execution context are included, ensuring deleted or obsolete tests do not skew the data.

#### **2. The Savings Formula**

The percentage of savings is the primary KPI for the efficiency of your optimization policy.

**Time Savings Percentage:** The formula calculates the ratio of skipped time relative to the total potential time:

$$
\text{% Savings (Time)} = 1 - \left( \frac{\text{Actual Time}}{\text{Actual Time} + \text{Skipped by SeaLights Time}} \right)
$$

**Count Savings Percentage:** The same logic applies to the volume of tests:

$$
\text{% Savings (Count)} = 1 - \left( \frac{\text{Actual Test Count}}{\text{Actual Test Count} + \text{Skipped by SeaLights Count}} \right)
$$

#### **3. Defining Saved Time**

SeaLights is strict about what constitutes "Savings." **Saved Time** only accounts for the time of tests that met two criteria:

1. They were officially recommended to be skipped by the SeaLights analysis.
2. They were actually skipped (not executed) in the build.

***

### FAQ

<details>

<summary><strong>Why doesn't "Saved Time" always equal "Original Time - Actual Time"?</strong> </summary>

"Saved Time" is specifically tied to SeaLights' recommendations. If a test was skipped for other reasons (e.g., manually disabled in code or failed due to environment issues before the test started), it is not credited to SeaLights Test Optimization savings.

</details>

<details>

<summary><strong>How does SeaLights handle new tests?</strong></summary>

New tests have no historical data. SeaLights typically defaults to running new tests to establish a baseline and ensure coverage, meaning they will contribute to "Actual Time" and "Actual Test Count" in their first run.

</details>

<details>

<summary><strong>What happens if a test's execution time fluctuates wildly?</strong></summary>

SeaLights uses a rolling average of the last five runs to smooth out environmental noise and provide the most accurate "Original Time" estimate possible.

</details>