How SeaLights Builds Test Recommendations for SAP

SeaLights generates test recommendations for SAP manual testing by maintaining a mapping between SAP executions, the underlying SAP code objects, and the tests in your test suite. The quality and precision of recommendations depend on how deeply SeaLights is integrated into your testing workflow - specifically, whether test executions are reported back to SeaLights. There are three integration levels, each producing progressively more targeted recommendations.

The Two Models Behind Recommendations

Before describing the integration levels, it helps to understand the two models SeaLights uses to generate recommendations:

Semantic Model

SeaLights maps SAP executions to their associated SAP code objects, and links those objects to the tests in your test repository. When a transport is applied, SeaLights identifies which objects changed and recommends the tests mapped to those objects. This model works from day one, with no execution history required, but because it relies on structural mapping rather than observed behavior, it may produce a broader recommendation list than strictly necessary.

Statistical Model

As test executions are reported back to SeaLights, the system learns which tests actually exercise which code paths. Over time, this observed data refines the semantic mapping, improving precision and reducing the recommendation list to only the tests most likely to catch a regression. The statistical model builds on top of the semantic model - it does not replace it.

Every integration level starts with the semantic model. The statistical model develops progressively as execution data is reported to SeaLights, improving recommendation effectiveness over time.

The Three Integration Levels

Level 1: Full Integration

SeaLights is connected directly to your test management tool. Recommendations are delivered inside the tool, and test execution results are automatically reported back to SeaLights.

This is the most effective integration level. Because SeaLights receives execution data continuously, the statistical model develops quickly and recommendations become highly targeted. The SeaLights cycle UI is not needed, the full workflow happens within the test management tool.

Best for: Teams whose test management tool supports a native SeaLights integration and want a fully embedded experience with minimal process change.

Level 2: Semi Integration

Test execution results are reported to SeaLights, but recommendations are retrieved via the SeaLights UI rather than through a test management tool integration. Testers use the Test Recommendations page in SeaLights to create cycles, download recommended test lists, and run them in their tool of choice.

Because execution results are still reported back, the statistical model builds over time, and Carry Over cycles become especially effective. When a new cycle is created from a previous one, SeaLights automatically excludes tests that were already executed and passed, so teams continue exactly where they left off.

Best for: Teams that report execution results to SeaLights but do not have a full test management tool integration.

Level 3: Semantic Integration

Test execution results are not reported to SeaLights. SeaLights works purely from the structural mapping: executions, their associated SAP objects, and the linked tests in the external repository. This mapping is kept current as transports are applied and as the test repository evolves.

Recommendations are generated entirely from the semantic model. Without execution data, the statistical model cannot be built, so recommendations may be broader than in higher integration levels. There is also no basis for automatic Carry Over exclusion of completed tests.

Best for: Teams that cannot or do not report execution results to SeaLights, but still want change-driven test recommendations based on transport impact.

Supported Test Management Frameworks

SeaLights can construct and maintain the semantic mapping, between SAP executions, ABAP objects, and test cases, for the following frameworks. This means teams using these frameworks can receive recommendations via the SeaLights UI based on the semantic model.

qTest (Cloud and on-prem)
Tosca (Cloud and on-prem) Although Tosca is an automated testing framework, teams can use this flow to get recommendations from the semantic model, then execute the recommended tests through their regular Tosca integration, with results reported back to SeaLights as normal. This allows them to benefit from targeted recommendations even before a statistical model has been built.
OpenText ALM
Worksoft Certify
SAP Solution Manager
Microsoft Azure DevOps
Xray (Requires Jira export)
Excel

Comparison Summary

Level 1 - Full

Level 2 - Semi

Level 3 - Semantic

Recommendations delivered via

Test management tool

SeaLights UI

Execution results reported to SeaLights

Yes (automatic)

Yes

Semantic model

Statistical model

Builds over time

Not available

Carry Over effectiveness

High

Limited

Recommendation precision

High

Moderate - broader lists expected

SeaLights cycle UI required

Yes

FAQ

What does the recommendation list look like on day one, before any execution data exists?

All integration levels start from the semantic model, so initial recommendations are based purely on structural mapping. Expect a broader list at first. As execution data accumulates (Level 1 and 2), precision improves automatically, no reconfiguration is needed.

How long does it take for the statistical model to meaningfully improve recommendations?

The model improves incrementally with each reported execution cycle, each test executed will have better recommendations in future runs.

Can we start at Level 3 and move to a higher integration level later?

Yes. The semantic model is always active for the supported frameworks, so moving to Level 2 or Level 1 later means execution data begins accumulating from that point forward. Historical executions that were not reported cannot be retroactively added.

Is the Carry Over feature useful at Level 3?

Carry Over is available at all levels, but its automatic exclusion of completed tests relies on execution results being reported to SeaLights. At Level 3, without that data, the exclusion logic has no basis to work from, meaning Carry Over will not automatically narrow the list based on what was already tested.

Who defines and maintains the semantic mapping between executions, SAP objects, and tests?

The initial mapping is established during onboarding. SeaLights then keeps the mapping current automatically as transports are applied. Changes to the test repository structure may require periodic review.

I created a cycle by mistake. What should I do?

Simply create a new cycle with the correct configuration. This will automatically close the mistaken cycle and replace it as the active one. Note that if your intention was to keep the previous cycle (the one that was active before the mistake) open, that is not currently possible, once a cycle is closed by creating a new one, it cannot be reopened.

Was this helpful?