> For the complete documentation index, see [llms.txt](https://docs.sealights.io/knowledgebase/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.sealights.io/knowledgebase/knowledge-base/sealights-for-abap/how-it-works/day-zero-structural-analysis-no-execution-history-required/building-the-optimal-test-set.md). # Building the Optimal Test Set ### High-Level Description #### Core Capability The Optimal Test Set is the final output of Day Zero — the smallest possible combination of tests from the connected repository that guarantees every Most-At-Risk executable is exercised at least once. It is the answer to the question: *what is the minimum we must run to cover the risk?* Alongside it, SeaLights also produces the full covering test list — every test that exercises at least one most-at-risk executable — giving teams the choice between minimum scope and maximum confidence. #### Interface Experience and Navigation Both outputs are accessible in the **Tests to Run** tab of any active cycle. Three metrics summarize the recommendation: **Potential Test Savings** (the reduction from the full repository), **Impacted Tests** (the full covering list count), and **Optimal Tests to Run** (the minimum sub-set count). Both lists are downloadable via the **Download Test Recommendations** button, ready to hand directly to the test runner. *** ### Business Value & Key Use Cases * **Replacing full-suite regression runs** — Teams no longer need to run thousands of tests on every transport import. The optimal set reduces execution scope to what actually covers the risk — in observed cycles, by up to 96% — while leaving no most-at-risk executable unverified. * **Objective scope decisions under time pressure** — The optimal set removes subjectivity from the question of how much to test when time is constrained. The answer is always: run the optimal set and every most-at-risk executable is covered. * **Demonstrating testing efficiency** — The Potential Test Savings percentage is a cycle-specific, quantified metric that demonstrates the value of risk-based testing to QA leadership and release governance boards. #### Impact Metrics In a repository of 5,000+ tests, the optimal set for a given cycle typically contains dozens to low hundreds of tests — enough to cover every most-at-risk executable, no more. The full covering list is larger but still a fraction of the full repository. Every test outside the covering list is irrelevant to this cycle's specific risk profile. *** ### Detailed Functionality #### Covering Tests vs. Optimal Tests Both concepts operate on the same pool of Most-At-Risk executables, but they answer different questions: **A covering test** exercises at least one most-at-risk executable. It is genuinely relevant to this cycle's risk. However, other covering tests may exercise the same executable — making some tests redundant relative to others when only minimum coverage is needed. **An optimal test** belongs to the minimum combination that still guarantees every most-at-risk executable is hit at least once. The optimal set eliminates redundancy: no executable is left uncovered, and no test in the set is superfluous. ``` Full Repository (e.g. 5,094 tests) │ ▼ Covering Tests (e.g. 2,093) — every test that hits at least one most-at-risk executable │ ▼ Optimal Test Set (e.g. 167) — the minimum subset that covers every most-at-risk executable ``` #### Choosing Which to Run | Situation | Run | | ------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | | Standard release, adequate time | Full covering list — every most-at-risk executable is hit, many by multiple tests | | Time-constrained release window | Optimal set — guaranteed full coverage of every most-at-risk executable, minimum execution time | | Risk-sensitive release (upgrade, regulated environment) | Full covering list, or extend to the full repository for maximum certainty | | First cycle with no execution history | Full covering list — broader execution builds the history that refines future recommendations | Running more tests than the optimal set never causes a problem. Any test that executes and reports back to SeaLights contributes to coverage and to the execution history that improves future cycles. #### How Tests Are Mapped to Executables SeaLights connects tests to most-at-risk executables using two sources of information, combined: **Structural links (Day Zero)** — Based on the same dependency graph used in Steps 1 and 2. If a test is associated with an executable in the test repository, and that executable depends on a changed object, the test is a candidate covering test. **Observed links (Day One Onward)** — From prior cycles where tests have already executed. SeaLights records which ABAP objects each test actually touched at runtime. These confirmed links are more reliable than structural predictions and take precedence when both exist. On the first cycle — before any execution history exists — the optimal set is built entirely from structural links. As execution history accumulates, confirmed links replace structural predictions, and the optimal set becomes progressively more accurate. #### Tests Not in Either List Tests that appear in the connected repository but do not appear in either the covering list or the optimal set have no link — structural or observed — to any most-at-risk executable for this cycle. They are not relevant to this cycle's specific risk. They may be relevant to other executables, other cycles, or future transport sets — they are not discarded from the repository, only excluded from this cycle's recommendations. *** ### FAQ **Q: The optimal set seems large relative to the number of most-at-risk executables. Why?** A: A large optimal set typically means the most-at-risk executables have low test overlap — each executable is covered by a different, non-overlapping test, so many tests are needed to achieve full coverage. This is common when the changed objects span many unrelated functional areas, or when the test suite has limited cross-functional tests. It is accurate, not a miscalculation. **Q: Can we run a subset of the optimal set and still be considered covered?** A: Running fewer tests than the full optimal set means at least one most-at-risk executable will be unverified in this cycle. That is a risk acceptance decision, not a coverage decision. If time constraints make this unavoidable, the Test Gaps tab will reflect which executables remain unverified. **Q: Does the optimal set change as execution history builds?** A: The optimal set for a given cycle is fixed at cycle creation. Execution history improves the optimal set for *future* cycles by replacing structural predictions with confirmed links. Over multiple cycles, this typically results in a tighter, more accurate optimal set. **Q: Tests we know should be in the recommendation aren't appearing. What should we check?** A: First, confirm the test exists in the connected test repository and that the repository connection is active. Second, check whether the test has a structural or observed link to a most-at-risk executable in this cycle. If the test covers an executable that is impacted but not ranked as most-at-risk, it will not appear in the recommendation. If the executable should be most-at-risk, consider adding it to the business-critical list to ensure it is always prioritized. *** *Part of the* [*Day Zero: Structural Analysis*](https://claude.ai/chat/how-01-day-zero.md) *section | SeaLights for ABAP* --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.sealights.io/knowledgebase/knowledge-base/sealights-for-abap/how-it-works/day-zero-structural-analysis-no-execution-history-required/building-the-optimal-test-set.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.