# Automated Testing Optimization

This playbook shifts automated testing from a "run-all" default to a precision model that balances instant feedback with comprehensive system validation. By **decoupling immediate change verification from extended use-case coverage**, we ensure every code modification is verified via the most efficient path, while deferring broad, repetitive validation to a lower frequency.

The objective is to make execution time strictly proportional to the risk and scope of a change. This framework provides the logic to calibrate test-skipping based on performance data, following one core directive: **optimization aggressiveness must be inversely proportional to release frequency** to maintain maximum velocity without compromising quality.

## The Inverse Law & Full-Run Strategy

The relationship between your release frequency, your testing strategy and your safety requirements.

<table><thead><tr><th width="136.125">Level Profile</th><th width="171.36328125">Release Frequency</th><th width="197.15234375">Aggressiveness</th><th>Recommended Full Run</th></tr></thead><tbody><tr><td><strong>Conservative</strong></td><td><strong>High:</strong> <br>Daily / Hourly / CD</td><td><p><strong>None/Low:</strong> </p><p>Run <em>all</em> impacted tests.</p></td><td><strong>Monthly/Optional:</strong> Entire test suite, only as a "sanity check".</td></tr><tr><td><strong>Moderate</strong></td><td><p><strong>Medium:</strong> </p><p>Weekly / Twice-Weekly</p></td><td><p><strong>Moderate:</strong> </p><p>Prioritize high-risk impacted areas.</p></td><td><strong>Weekly:</strong> Skipped impacted tests only, every weekend or Friday night.</td></tr><tr><td><strong>Aggressive</strong></td><td><p><strong>Low:</strong></p><p>Bi-Weekly / Monthly / Quarterly</p></td><td><p><strong>High:</strong> </p><p>Run only the most critical "smoke" impacted tests.</p></td><td><strong>Monthly/Bi-Weekly:</strong> Skipped impacted tests only, every sprint/month.</td></tr></tbody></table>

## Strategy Implementation

#### **Level 1: The Conservative Approach**

* **Logic:** Changes are small and frequent. Running all impacted tests usually takes very little time because the "delta" is minimal.
* **Risk:** Skipping even one impacted test increases the risk of "CD Churn", where a fix for one deploy breaks the next one an hour later.
* **Full Run Guidance:** Since you are already running every test relevant to your code changes, the risk of a "leak" is extremely low. You do not need an automated full run. If you want one just to be safe, schedule it **once a month** during off-hours to ensure no environment drift has occurred.

#### **Level 2: The Moderate Approach**

* **Logic:** Changes are larger. Running all impacted tests might start to delay the pipeline. Prioritization must be implemented.
* **Risk:** Over-aggressive skipping can lead to "Silent Regressions". Bugs that break the master branch integration when performing extended use-case coverage.
* **Full Run Guidance:** A **Weekly Full Regression** is recommended, including only the skipped impacted tests. This ensures that any use case affected by the code changes and was skipped during the week is validated before the next week starts.

#### **Level 3: The Aggressive Approach**

* **Why:** Monolithic structures or huge release cycles mean thousands of tests are "impacted". You cannot run them all without killing productivity.
* **Risk:** Over-aggressive skipping can lead to "Integration Blindness". Bugs that break the master branch integration when performing extended use-case coverage.
* **Full Run Guidance:** It is recommended to run a **Monthly/Bi-weekly Full Regression** (Nightly), including only the skipped impacted tests. This ensures that any use case affected by the code changes and was skipped during the sprint/month is validated before the next sprint/month starts.

## Strategy Adjustments

Optimization is not a "set it and forget it" task. You must adjust based on outcomes.

#### **Scenario A: Velocity is great, but bugs are leaking to Production.**

1. **Analyze the Leak:** Use SeaLights to see if the bug occurred in a "Skipped Impacted" test. If yes, consider one of the next suggestions.
2. **Adjust Aggressiveness:** Move one level to the left (e.g., from Aggressive to Moderate).
3. **Increase Frequency:** If you are at Level 2, move your Full Run from Weekly to Twice-Weekly.
4. **Targeted Hardening:** If the leaks are always in a specific module (e.g., Payment Gateway), set a rule to *never* skip tests that cover critical use cases in that specific area, regardless of the global setting.

#### **Scenario B: Savings are high, but the team feels the "Nightly" is always broken.**

1. **Diagnostic:** This means your daytime skipping is *too* aggressive. Developers are checking in code that works in isolation but breaks the system.
2. **Solution:** Perform one of the following:
   1. **Adjust Aggressiveness:** Move one level to the left (e.g., from Aggressive to Moderate).
   2. **Increase Frequency:** Use **Intermediate Runs.** Introduce a "Mid-day/week/month Full Run" to catch errors before the end of the day/week/month.

#### **Scenario C: Safety is 100%, but Build Times are creeping up.**

1. **Diagnostic:** You are likely being too conservative for your current growth.
2. **Solution:** Move one level to the right. If you were running all impacted tests, try implementing the Moderate level or Aggressive Level.
3. **Cleanup:** Use SeaLights data to identify redundant tests by analyzing one of the test cycles that had low savings. Remove the redundant tests from your test suite.
4. **Customized Optimization:** Work with your SeaLights customer success manager to identify if you have common code that triggers too many of your tests, or if you have too many failed tests that you run over and over again without changing anything in the code. They will instruct you how to customize your strategy to handle such use cases.  &#x20;

## Success Metrics to Watch

* **Test Optimization Efficiency:**&#x20;
  * Calculation: Tests Skipped / Total Tests
  * Target:  > 60% for Level 3.
* **Slippage Rate:**&#x20;
  * Calculation: Bugs found in Full Run / Bugs found in TIA Run
  * &#x20;Target: Should stay below 5%.
* **Cycle Time:**&#x20;
  * Calculation: Average time from Commit to Test Result
  * Target: Should align with your Level's goals.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.sealights.io/knowledgebase/guides/test-optimization/test-optimization-playbooks/automated-testing-optimization.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.