Quality Gates

SeaLights Quality Gate feature allows you to define thresholds for build quality and ensure that only builds meeting your standards are promoted. Quality Gates act as checkpoints, evaluating builds against specific criteria and providing a clear indication of whether a build is ready for release or requires further attention.

Quality Gate Parameters

Quality Gates evaluate builds based on three key parameters:

• Code Changes Coverage (Mandatory): The percentage of code changes covered by tests. This metric helps ensure that recent modifications are adequately tested. You can specify minimum coverage for all test stages or for specific ones.

• Overall Code Coverage (Optional): The percentage of the entire codebase covered by tests. This metric provides a holistic view of test coverage across the application/component. You can specify minimum coverage for all test stages or for specific ones.

• Failed Tests (Optional): This parameter offers two options for defining acceptable failure thresholds:

  • No Failures (Strict): The Quality Gate fails if any tests fail. This is a stringent requirement suitable for projects where zero failures are the target. You can apply this to all test stages or specific ones.

  • Percentage of Failures (Flexible): Allows you to define a threshold for acceptable failure rates. The Quality Gate will pass if the percentage of failed tests is below the defined value. This option provides more flexibility and can be tailored to projects with varying risk tolerance. You define the acceptable percentage. You can apply this to all test stages or specific ones.

Configuring Quality Gates

Quality Gates can be configured at different levels:

• SeaLights Default: SeaLights provides default settings for Quality Gates. These are automatically applied as the default for new customer accounts.

• Customer Default: Customers can edit the SeaLights default settings to create their own customer default. This customer default can then be applied to all applications.

• Application-Specific: It's possible to define different Quality Gates for individual applications. These settings override the customer default. Applications can revert to the customer default or the SeaLights default at any time.

Management

• Default Settings (SeaLights & Customer): Managed from Settings / Quality Gates page.

• Application Quality Gate Settings: Managed from the Coverage Dashboard, the Coverage Report, or Settings / Quality Gates page.

Failed Test Counting Methodology

SeaLights offers two methods for counting failed tests:

• All Results Method (Current Behavior): Counts all test runs, even if subsequent runs of the same test pass. This can inflate the perceived number of failures.

• Latest Result Method (New): Counts only the latest test run result for each unique test. This provides a more accurate representation of actual failures.

Example

Test	Run 1	Run 2
A	Passed	Passed
B	Failed	Passed
C	Failed	Failed
D	Failed	Failed

• All Results Method: 5 failures out of 8 test runs.

• Latest Result Method: 2 failures out of 4 tests.

Note: Skipped tests are not counted in either method.

Configuration

This setting is currently managed manually with the help of your SeaLights customer success representative. Changing this setting will likely impact the calculated percentages of failed tests and potentially the Quality Gate status of existing builds.

Understanding Quality Gate Status

Each build is evaluated against the defined Quality Gate criteria, resulting in one of the following statuses:

• Passed: All Quality Gate conditions were met. The build successfully satisfies the predefined quality thresholds.

• Failed: The code did not meet one or more quality thresholds. The build failed to meet the minimum requirements defined in the Quality Gate.

• Missing Data: The Quality Gate could not be fully evaluated due to one or more of the following:

  • No Reported Tests

  • Missing Test Stage Data

  • No Detected Code Changes

• Scan Issue: The build scan encountered errors during the reporting process. This indicates a problem with the scan itself, rather than the code quality. Investigate the scan logs to identify the root cause of the issue.

Quality Gate Status Display

Quality Gate Status

The Quality Gate status is displayed on the Coverage Dashboard and Coverage Report. Clicking on the status opens a pop-up window summarizing the build status and Quality Gate definitions, including:

• Overall Quality Gate status.

• Defined thresholds for each parameter.

• Actual values measured for each parameter.

• Clear indication of which criteria were met and which were not.

Identifying Code Changes

By default, each build is compared to its previous build to identify code modifications. You can define a reference build (any prior build). In this case, code modifications between the current build and the reference build are considered. For example, you can select the production release build as the reference build to ensure all changes since the last release are covered by tests in the latest build.