Manual Testing Optimization

This playbook shifts manual testing from a "full regression" default to a precision model. By decoupling immediate change verification from extended manual validation, we ensure every code modification is verified via the most efficient path, while strategically managing the "Impacted & Skipped" pool over the course of a release cycle.

The objective is to make manual execution time strictly proportional to the risk and scope of a change. This framework provides the logic to calibrate manual test-skipping based on SeaLights impact data, following one core directive: Optimization must be used to front-load high-risk testing, ensuring the "Impacted & Skipped" pool is systematically depleted as the release date approaches.

The Dynamics of the The "Impacted & Skipped" Pool

In manual testing, the dynamism of the "Impacted & Skipped" is crucial:

1. Growth (The Delta): As developers push new code, the pool of impacted tests grows. However, because manual tests (E2E/Regression) cover large logic blocks, the pool often stabilizes because the same high-level tests are triggered by multiple code changes.

2. Depletion (The Natural Clearance): Every week, as manual testers execute tests for new features or bug fixes, they often touch areas that were previously in the pool.

3. The Convergence: By the end of a release cycle, the pool should naturally shrink. The goal is to ensure that by "Release Week," the pool of "Impacted but never run" tests is small enough to be cleared in a final validation pass.

CRITICAL: Reporting Tests to SeaLights

The efficiency of this model depends entirely on the feedback loop. If manual tests are not reported back to SeaLights, the optimization becomes inefficient over time.

1. Daily Reporting Cadence: It is vital to report what has been tested at least once a day. This allows SeaLights to continuously improve its statistical modeling and refine the code-to-test map. Regular reporting ensures that recommendations become more accurate over time as the system "learns" which tests effectively cover specific code changes.

2. The "Ghost Pool" Problem: If a tester manually runs an impacted test but doesn't report it, SeaLights assumes the test is still "Skipped". SeaLights will continue to recommend the same tests for the next code change, even if they were just verified, leading to redundant work.

3. Tester Specificity (Lab IDs): Where possible, assign a specific Lab ID per tester. Reporting the unique Lab ID alongside test results allows for granular tracking of testing progress across the team and prevents data collisions.

4. Result Transparency: Ensure "Failed" manual tests are recorded immediately reflect accurate status of quality gate on Coverage Dashboard. Also, it allows preventing the optimization engine from suggesting the same tests for next cycles.

The Inverse Law & Manual Full-Run Strategy

The relationship between your manual testing capacity, your release frequency, and your safety requirements.

Strategy	Release Frequency	Optimization Level	Recommended Full Run
Conservative	High: Daily / Weekly	Low: Run all impacted manual tests.	Monthly/Optional: Full manual suite only as a periodic sanity check.
Moderate	Medium: Bi-Weekly / Monthly	Moderate: Prioritize high-risk impacted areas in the sprint.	Weekly: Run the "Impacted & Skipped" pool every second week.
Aggressive	Low: Quarterly / Large Releases	High: Run only critical "Smoke" impacted tests per feature.	Monthly/Sprint-End: Dedicated "Hardening Sprints" to clear the Skipped Pool.

Strategy Implementation

Level 1: The Conservative Approach

• Logic: Changes are small. The "delta" triggers a manageable number of manual scripts.

• Action: Execute 100% of impacted tests within the same cycle as the code change.

• Risk: Low. Minimal risk of "Leakage" because the manual verification keeps pace with development.

Level 2: The Moderate Approach

• Logic: Changes are larger. Running all impacted tests manually would delay the sprint.

• Action: Focus on the most critical impacted tests. Other impacted tests are stored in the "Impacted & Skipped" pool.

• Full Run Guidance: A Bi-Weekly Catch-Up is recommended. Testers spend every second week (or part of it) running tests that were "Impacted & Skipped" during the previous week.

Level 3: The Aggressive Approach

• Logic: Thousands of manual tests are impacted by deep architectural changes.

• Action: Run only the "Impacted Smoke" tests. The majority of impacted tests enter the "Impacted & Skipped" pool to maintain development velocity.

• Full Run Guidance: Use Bi-Weekly/Monthly Hardening. The manual team systematically "chips away" at the Skipped Impacted pool so that the "Risk Debt" does not explode at the end of the quarter.

Strategy Adjustments

Scenario A: The "Skipped Pool" is growing too fast.

• Diagnostic: You are skipping more than you are clearing.

• Solution: Increase "Catch-Up" frequency. Move from Monthly to Bi-Weekly full runs. Ensure all manual runs are being reported to SeaLights correctly.

Scenario B: Manual Build Times are high, but Savings are low.

• Diagnostic: You are likely being too conservative.

• Solution: Move one level to the right (e.g., Moderate to Aggressive). Use SeaLights data to identify "Over-lapping" tests that cover the same code and retire redundant manual scripts.

Success Metrics to Watch

• Manual Optimization Efficiency:

  • Calculation: Manual Tests Skipped / Total Manual Suite

  • Target: > 50%.

• Reporting Compliance:

  • Calculation: % of manual sessions successfully integrated/uploaded to SeaLights.

  • Target: 100%.