Ignore Failed Tests Policy

The Ignore Failed Tests capability is a specialized filter used during the test recommendation process to reduce noise and focus on new code changes. It prevents the system from repeatedly recommending tests that are already known to be failing if the underlying code remains unchanged.

Why It Is Useful

In a fast-moving CI/CD environment, some tests may fail due to known bugs or environment issues that are not being addressed in the current sprint. Repeatedly running these failing tests wastes resources and can obscure new regressions. This logic allows teams to temporarily bypass these "noisy" failures during the optimization phase.

Detailed Functionality

• API-Driven Logic: This capability is not a static dashboard setting. It is triggered via the Test Optimization API when requesting test recommendations.

• Change-Based Validation: The logic checks two conditions:

  1. Did this test fail in the previous run?

  2. Has the code covered by this test changed since that failure?

• Exclusion Rules: If the test failed previously and the associated code has not changed, the test is excluded from the current recommendation list. If the code has changed, the test will be recommended regardless of its previous status to verify the fix.

Implementation

When asking for test recommendations through the API, users can toggle this filter. If a test failed in a previous run but the code it covers has not been modified in the current build, the API will exclude it from the recommended list.

---

FAQ

Does this mean I will never see these failures again?

No. The tests are only skipped by the optimization engine for that specific run. You can still view the original failures in the SeaLights dashboard.

Is this the same as "Quarantining" a test?

Not exactly. Quarantining usually hides a test entirely. This logic is smarter because it will automatically "un-hide" and recommend the test as soon as someone modifies the code it covers.