LogoLogo
Product
  • Knowledge Base
  • What's New
  • Guides
  • User Story Coverage
    • Getting Started
    • User Story Challenges & Solution
      • Typical Implementation
      • The Challenges
      • The Solution
    • User Story Coverage Report Overview
      • Release Go / No Go Report
        • How to Generate / Edit the Report
      • User Story Quality Overview
        • How to Generate the User Story View
      • User Story Coverage Analysis
        • How to Generate the Analysis View
      • Uncovered Methods View
        • How to Generate the View
      • Customization
      • Integration
    • Use Cases by Persona
      • Managers
        • Informed Go/No Go Decisions Making
        • Effective Resources Prioritization
        • Overall Progress Monitoring
      • Developers
        • Code Quality Ownership
        • Seamless Collaboration with QA
        • Code Review Facilitator
      • QA Engineers
        • Test Execution Progress Monitoring
        • Testing Effort Prioritization
        • Testing Strategy Planing
    • Technical Overview
      • User Story Coverage Mechanism
      • Technical Architecture
      • Deployment Guide
        • US1_getResults.sh
        • US2_createReport.sh
        • US_UpdateConfluence.sh
  • Test Optimization
    • Getting Started
    • Test Execution Challenges & Solution
      • The Challenges
      • Test Optimization Solution
      • Test Optimization Main Advantages
    • Test Optimization Overview
      • Automated Test Optimization
      • Manual Test Optimization
      • Test Optimization for Pull Request
      • Test Selection Policies
        • Full Run Policy
        • No Code Changes Policy
        • Common Code Policy
        • Fastest Path to 100% Coverage Policy
      • Integrations
    • Use Cases by Persona
      • Managers
        • Fast Delivery
        • Resource Optimization
        • Thorough Testing in Tight Schedule
      • Developers
        • Exploring Only Relevant Test Failures
        • Faster Feedback Loop
        • Shift Left Testing
      • QA Engineers & Manual Testers
        • Faster & Focused Manual Testing
        • Optimizing Test Suite
        • Having Stable Product for Testing
    • Technical Overview
      • Test Optimization Mechanism
        • Associating Code With Tests
          • Statistical modeling
          • One-to-One Mapping
          • Calibration
        • Detecting Modified Code
        • Generating Test Recommendations
      • Technical Architecture
      • Deployment Guide
  • Quality Improvement
    • Getting Started
    • Challenges & Approach Comparison
      • The Challenges
      • Quality Improvement Approaches
      • Choosing the Right Approach
    • Quality Improvement Solution Overview
      • Test Gaps Analysis Report
        • How to Generate / Edit the Report
      • Coverage Trend Report
        • How to Generate / Edit the Report
      • Proof of Testing Report
        • How to Generate / Edit the Report
      • Release Quality Improvement Guide
        • STEP 1: Deploy SeaLights
        • STEP 2: Take a Quality Snapshot
        • STEP 3: Prioritize Code Areas
          • Add Code Labels
          • Ignore Irrelevant Code
          • Perform a Deep CSV Analysis
        • STEP 4: Set Baseline & Threshold
        • STEP 5: Analyze Test Gaps
        • STEP 6: Write Tests
        • Step 7: Make a Go / No Go Decision Based on Quality Gate
        • STEP 8: Measure Defect Escape Rate
      • Over Time Quality Improvement Guide
        • STEP 1: Deploy SeaLights
        • STEP 2: Take a Quality Snapshot
        • STEP 3: Prioritize code areas
          • Add Code Labels
          • Ignore Irrelevant Code
          • Perform a Deep CSV Analysis
        • STEP 4: Set Baseline & Goal
        • STEP 5: Set timeline
        • STEP 6: Write tests
        • STEP 7: Monitor progress
        • STEP 8: Measure Defect Escape Rate
    • Use Cases by Persona
      • Managers
        • Effective Prioritization & Budget Allocation
        • Tracking Progress & Measuring Impact
        • Data-Driven Release Decisions
        • Transparency & Communication
      • Developers
        • Mastering Code Coverage
        • Seamless Collaboration with QA
        • Code Quality Ownership
      • QA Engineers
        • Prioritizing Test Efforts
        • Contributing to Release Informed Decisions
        • Seamless Collaboration with Developers
        • Evaluating Testing Strategy
    • Technical Overview
      • Solution Mechanism
      • Technical Architecture
      • Deployment Guide
  • Value Proposition
    • Overview
    • Quality Use Cases
      • Go/No Go Decisions
      • Quality Improvement & Test Gaps
      • Governance & Quality Gates
      • Compliance & Proof of Testing
    • Test Optimization Use Cases
      • Reduce Costs & Infrastructure
      • Shorten Release Cycles
      • Reduce Troubleshooting
Powered by GitBook
On this page
  • Test Optimization Summary Page
  • App List
  • Test Stage Breakdown
  • Build Breakdown Page
  • Overview
  • Builds

Was this helpful?

  1. Test Optimization
  2. Test Optimization Overview

Automated Test Optimization

PreviousTest Optimization OverviewNextManual Test Optimization

Was this helpful?

SeaLights Test Optimization simplifies your testing process by analyzing code changes and recommending which tests are directly impacted. Here's how it works:

  1. Automated Test Reporting: SeaLights automatically collects data from your tests through integrations with various testing frameworks or using its APIs.

  2. Test Optimization Engine (TIA): Starting from the second build, TIA analyzes code changes and historical data to identify tests impacted by modifications.

  3. Recommendations and Execution: TIA recommends skipping or prioritizing tests based on their relevance and according to a pre set . These recommendations are automatically applied and executed through integrations or by using the APIs.


Test Optimization Summary Page

App List

  • List all apps with builds and testing activity within your selected timeframe.

  • TIA Status: On, Partially On, Off. Partially On means only some of the test stages are executed with TIA test selection On.

  • Compute Time & Savings per app. Displays total actual execution data and savings per app, aggregating test run time across all the machines.

  • Potential Results with TIA ON: Shows in brackets estimated savings and optimized test compute time, in case TIA was On.

Each app is clickable to view the breakdown of the test stages.

Test Stage Breakdown

  • List of all test stages which ran against at least 1 build included in the selected date range.

    • Exception! All manual test stages included in the list, even if no manual test was executed yet, in order to provide manual testers with test recommendations.

    • Test stages that are still in progress are not included in the list.

    • Unsupported testing frameworks will not receive recommendations (full test runs will occur).

  • TIA Status: On, Off.

  • Builds: Number of builds the test stage was running against.

  • Number of Tests: Total number of tests executed across all builds. This is the accumulation of test runs, and not unique tests. In addition, average number of test executed per build.

  • Compute Time & Savings: Actual execution data and savings across all builds and an average per build.

  • Potential Results with TIA ON: Shows in brackets estimated savings and optimized test compute time, in case TIA was On.

Test recommendations prioritize the highest testing standards, encompassing your entire codebase, even if specific sections are marked for ignoring in other settings.

Build Breakdown Page

Overview

Looking at a specific app and test stage, you can view the total number of tests and compute time and savings, as well as average per build. Graphs displaying individual build data from the last 14 days are also available.

Builds

You can switch to the Builds view to view test execution data per build.

  • All tests with indication of ran / skipped and recommendation reason

  • Code to test list, including all code changes in the build and their corresponding tests.

Test Selection Policy
Test Optimization Summary
Test Stage Breakdown