Savings Calculation

Test Optimization, powered by SeaLights' Test Impact Analysis (TIA), provides valuable insights into the efficiency of your test cycles. The "savings" metrics you see in SeaLights are a direct reflection of how much time and effort were conserved by intelligently skipping non-impacted tests. This document explains the methodology behind these calculations.

The Core Concept

The calculation of Test Optimization savings is based on a simple comparison: the time and number of tests required for a full, traditional test run versus the time and number of tests in an optimized test run using TIA.

Calculating Compute Time Savings

A note on "Compute Time":

It's important to understand that "compute time" refers to the total machine time required to run tests, not the actual elapsed wall-clock time. In parallel testing environments, the actual run time might be shorter than the compute time because multiple tests are running simultaneously across different machines. Our calculation refers to the sum of all machine time spent.

To determine the savings in compute time, SeaLights performs the following steps:

1. Calculate "Compute Time Without TIA": This represents the total time it would have taken to execute every test in the build or test cycle.

   • This calculation only takes into account tests that were reported in the current execution, ensuring that deleted tests are not included.

   • For tests that actually ran (passed or failed) in the build, their real execution duration is used.

   • For tests that were skipped by TIA, their duration is estimated by calculating the average of their last five historical runs.

   • All these durations are summed together to get the total estimated time for a full, non-optimized run.

2. Calculate "Compute Time With TIA": This represents the actual time spent executing only the tests that were chosen to run by TIA.

   • Only the durations of the tests that were actually run are included in this total.

3. Determine the Savings: The final savings in compute time is the difference between these two totals.

   • Compute Time Savings = (Compute Time Without TIA) - (Compute Time With TIA)

Calculating Savings in Number of Tests

This metric is calculated in a similar, straightforward manner:

1. Total Number of Tests Without TIA: This is the total number of tests in the build or test cycle.

2. Number of Tests With TIA: This is the actual number of tests that were executed in the optimized run.

3. Determine the Savings: The savings in the number of tests is the difference between the total number of tests and the number of tests that ran.

   • Test Count Savings = (Total Number of Tests) - (Number of Tests that Ran)

Calculating the Savings Percentage

The percentage of savings is a key performance indicator that shows the efficiency of your TIA policy.

1. Percentage Calculation: The percentage is derived by dividing the number of tests saved by the total number of tests that would have run in a non-optimized scenario.

   • Savings Percentage = (Test Count Savings) / (Total Number of Tests Without TIA)