FAQ

Common questions about the User Story Coverage Tagging tool.

General Questions

What is this tool for?

This tool connects issue tracking systems (Jira, Azure DevOps) with SeaLights test coverage platform. It automatically tags code changes with ticket IDs and generates coverage reports at the user story level, helping teams track which features have adequate test coverage.

Do I need to install anything on my servers?

No. This is a Python command-line tool that runs wherever Python 3.7+ is available. It connects to your existing systems (SeaLights, Jira/ADO, GitHub) via their APIs.

Which platforms are supported?

Issue Trackers:

• Jira Cloud

• Jira On-Premises (with plugin)

• Azure DevOps

Source Control:

• GitHub

• GitLab

• Bitbucket

• Azure DevOps Git

Coverage Platform:

• SeaLights

How much does it cost?

The tool itself is free (part of SeaLights integration). You need existing subscriptions to:

• SeaLights (for test coverage data)

• Your issue tracker (Jira or Azure DevOps)

• Your source control platform (GitHub, GitLab, etc.)

---

Setup and Configuration

How do I get started?

1. Install Python 3.7+ and the requests library

2. Copy settings.sample to settings

3. Fill in your credentials and configuration

4. Run the Quick Start example

See the Quick Start Guide for a complete walkthrough.

Where do I get API tokens?

SeaLights:

• Log into SeaLights web UI

• Navigate to Settings → API Tokens

• Generate a new token

Jira Cloud:

• Go to https://id.atlassian.com/manage/api-tokens

• Create API token

• Use with your email: echo -n "email:token" | base64

GitHub:

• Go to https://github.com/settings/tokens

• Generate new token with repo scope

Azure DevOps:

• User Settings → Personal Access Tokens

• Create token with Work Items (Read) and Extension Data (Read & Write) permissions

Can I use environment variables instead of a settings file?

Yes! All settings can be overridden with environment variables. This is recommended for CI/CD pipelines.

export SEALIGHTS_API_TOKEN="your-token"
export JIRA_BASE_URL="https://yourcompany.atlassian.net"
python3 US1_RPT_create_report_info_files.py --settings settings

See Configuration for the complete list.

Can I use multiple settings files?

Yes. Specify multiple --settings flags to merge configurations:

python3 US1_RPT_create_report_info_files.py \
  --settings base_config \
  --settings team_config \
  --settings credentials

Settings are merged left to right, with later files overriding earlier ones.

---

Tagging Questions

How does the tool extract ticket IDs?

The tool uses regex patterns to extract ticket IDs from:

• Commit messages

• Branch names

• Pull request titles

Configure the pattern in settings:

{
  "scm": {
    "key_pattern_regex": "PROJ-[0-9]+"
  }
}

Do I need to tag code before generating reports?

It depends on your SeaLights configuration.

If your CI/CD pipeline reports builds to SeaLights with git commit information (commit messages, branch names), SeaLights may already be auto-tagging your code with ticket IDs using regex patterns configured by your Customer Success representative. If auto-tagging is enabled, you can skip the tagging scripts entirely and go directly to report generation.

If auto-tagging is not configured, yes—you need to run the tagging scripts first:

1. Tag Phase - Run US_SRC_* scripts to associate code changes with ticket IDs in SeaLights

2. Report Phase - Run US_RPT_* scripts to query SeaLights and generate reports

To check if you need tagging scripts:

• Ask your SeaLights Customer Success representative if auto-tagging is configured for your apps/branches

• Try generating a report—if tickets show 0% coverage despite having tests, tagging may be needed

What is SeaLights auto-tagging and how do I enable it?

SeaLights auto-tagging is a feature where SeaLights automatically extracts ticket IDs from git commit information (commit messages or branch names) when builds are reported to SeaLights. This eliminates the need to run the US_SRC_* tagging scripts.

How it works:

• Your CI/CD pipeline reports builds to SeaLights including git commit information

• SeaLights uses configured regex patterns to extract ticket IDs

• Code is automatically tagged with those ticket IDs

• You only need to run the report generation scripts (US_RPT_*)

How to enable it:

• Contact your SeaLights Customer Success representative

• Provide: ticket naming regex pattern (e.g., PROJ-[0-9]+) and scope (apps/branches)

• SeaLights CS will configure auto-tagging for you

• This is not a self-service feature

Benefits:

• No need to run tagging scripts

• Tagging happens automatically with each build

• Simpler workflow—just reporting phase

• No separate tagging infrastructure needed

How often should I run tagging?

It depends on your workflow:

Daily/Continuous:

• Run after PRs are merged

• Run nightly for all recent changes

Sprint-based:

• Run at sprint start to capture previous sprint

• Run mid-sprint for progress tracking

• Run at sprint end for final reports

On-demand:

• Before releases

• When specific reports are needed

Can I tag retroactively?

Yes! Use --since-date to tag historical changes:

python3 US_SRC_tag_repos_by_github_prs.py \
  --settings settings \
  --since-date 2026-01-01

What if I don't use pull requests?

Use the git history tagging script:

python3 US_SRC_tag_repos_by_history.py \
  --settings settings \
  --days-back 30

This analyzes commit history directly instead of PRs.

---

Reporting Questions

What formats are reports available in?

• JSON - Raw data files (ReportInfo_*.json)

• HTML - Interactive web pages

• Confluence - Wiki pages

• Jira/ADO Fields - Coverage percentages in issue tracker

Can I customize the HTML reports?

Not directly through configuration. The HTML reports are generated from templates embedded in the scripts. You can modify the scripts themselves to customize the output.

How long does report generation take?

Varies based on:

• Number of tickets (1-100 tickets: minutes, 100+ tickets: 10-30 minutes)

• Number of apps/branches

• SeaLights API response time

• coverage_days_back setting

Typical: 5-15 minutes for 20-50 tickets.

Can I speed up report generation?

Yes:

• Reduce coverage_days_back

• Process fewer tickets (narrow JQL/WIQL)

• Use --report-id to reuse existing coverage report

• Process tickets in batches

What if a ticket has no coverage data?

The report will show 0% coverage. This usually means:

• Code hasn't been tagged with this ticket ID

• No tests were run against the tagged code

• The lookback period doesn't include test runs

---

Integration Questions

Can I integrate this with CI/CD?

Yes! See Examples - CI/CD Integration for:

• GitHub Actions

• Jenkins

• GitLab CI/CD

• Azure Pipelines

Can I fail builds based on coverage thresholds?

Yes. Parse the generated JSON files and check coverage values:

coverage=$(jq -r '.overallResults.overall_coverage' reports/ReportInfo_PROJ-123.json)
if (( $(echo "$coverage < 70" | bc -l) )); then
  exit 1
fi

See Example 19.

Can I send notifications?

Yes. The tool outputs structured JSON that you can parse and send to:

• Slack

• Microsoft Teams

• Email

• Custom webhooks

See Example 20.

Does it work with monorepos?

Yes. Configure multiple entries in repo_list, each pointing to different paths or the same repo with different filters.

Can I track coverage across multiple repositories?

Yes. List all repositories in repo_list:

{
  "repo_list": [
    {"repoName": "frontend", "SealightsAppName": "frontend-app"},
    {"repoName": "backend", "SealightsAppName": "backend-api"},
    {"repoName": "mobile", "SealightsAppName": "mobile-app"}
  ]
}

---

Test Stage Questions

What are test stages?

Test stages represent different phases of testing (unit, integration, E2E, etc.). They must match the stage names in your SeaLights configuration.

How do I know what my test stages are called?

Check your SeaLights dashboard or Test Gap Analysis (TGA) reports. The stage names used there should match the name field in your settings.

What are grouped test stages?

Grouped test stages aggregate coverage from multiple individual stages. For example, "Overall" might combine Unit + Integration + E2E.

{
  "grouped_test_stages": [
    {
      "name": "Overall",
      "stages": ["Unit_Tests", "Integration_Tests", "E2E_Tests"]
    }
  ]
}

Can I add custom test stages?

Yes, as long as they exist in SeaLights. Add them to the test_stages array in settings.

---

Jira-Specific Questions

What JQL should I use?

Depends on your goal:

Current sprint:

sprint in openSprints() AND project = MYPROJ

Last 30 days:

project = MYPROJ AND created >= -30d

Specific epic:

"Epic Link" = MYPROJ-100

By labels:

project = MYPROJ AND labels = sprint-42

How do I find custom field IDs?

1. Open a ticket in Jira

2. View the ticket JSON:

   • Jira Cloud: https://yourcompany.atlassian.net/rest/api/3/issue/PROJ-123

   • Use browser dev tools or curl

3. Search for your field in the JSON

4. Note the customfield_XXXXX ID

Do I need admin access to Jira?

No, but you need:

• Read access to projects in your JQL

• Write access if updating custom fields (US4_RPT_update_jira.py)

• Plugin admin rights if using the Jira plugin

---

Azure DevOps-Specific Questions

What WIQL should I use?

User stories in last 30 days:

SELECT [System.Id] FROM WorkItems
WHERE [System.WorkItemType] = 'User Story'
AND [System.ChangedDate] >= @Today - 30

Specific iteration:

SELECT [System.Id] FROM WorkItems
WHERE [System.IterationPath] = 'MyProject\\Sprint 42'

By area path:

SELECT [System.Id] FROM WorkItems
WHERE [System.AreaPath] UNDER 'MyProject\\Team A'

How do I test my WIQL?

In Azure DevOps:

1. Go to Boards → Queries

2. Create new query

3. Switch to "Editor" mode

4. Paste your WIQL

5. Run and verify results

What permissions do I need in ADO?

• Work Items: Read

• Extension Data: Read & Write (if using ADO plugin)

Create a PAT with these permissions.

---

SeaLights Questions

Do I need a SeaLights agent running?

No. The tool uses the SeaLights REST API. However, you need:

• SeaLights agents running during your tests (to collect coverage)

• Builds reported to SeaLights with your app/branch names

What if my app name in SeaLights is different from my repo name?

That's fine and common. Map them in repo_list:

{
  "repo_list": [
    {
      "repoName": "my-git-repo",
      "SealightsAppName": "My Application Name in SeaLights"
    }
  ]
}

What are integration builds?

Integration builds are SeaLights builds that combine multiple component apps. Enable with:

{
  "coverage_app_list": [
    {
      "app_regex": "my-integration-build",
      "branch_regex": "main",
      "integration_build": true
    }
  ]
}

The tool will validate that all components exist in repo_list.

Can I use regex for app and branch names?

Yes! The app_regex and branch_regex fields are regex patterns:

{
  "coverage_app_list": [
    {"app_regex": "frontend-.*", "branch_regex": "main|master"}
  ]
}

---

Performance Questions

Why is tagging so slow?

Common causes:

• Large time range - Reduce --days-back

• Many PRs/commits - Process repos in parallel

• Large repositories - Git clone/pull takes time

• Network latency - Run closer to your git server

Why is report generation slow?

Common causes:

• Many tickets - Process in batches

• Long lookback - Reduce coverage_days_back

• SeaLights processing - First report takes longer, subsequent reports are cached

Can I process repositories in parallel?

Yes, use GNU Parallel or run multiple scripts simultaneously:

python3 US_SRC_tag_repos_by_github_prs.py --settings settings_repo1 &
python3 US_SRC_tag_repos_by_github_prs.py --settings settings_repo2 &
wait

---

Troubleshooting Questions

Where can I find error logs?

Enable debug logging and redirect to a file:

python3 US1_RPT_create_report_info_files.py \
  --settings settings \
  --log-level DEBUG \
  2>&1 | tee debug.log

The tool says "No tickets found" but I can see them in Jira/ADO

Verify:

1. Your JQL/WIQL returns results in the web UI

2. API credentials have read access to those tickets

3. Settings file has correct base URL and authorization

Test API directly:

curl -H "Authorization: Basic ..." \
  "https://yourcompany.atlassian.net/rest/api/3/search?jql=project=MYPROJ"

Reports show 0% coverage for all tickets

This means code hasn't been tagged. Run tagging scripts first:

python3 US_SRC_tag_repos_by_github_prs.py --settings settings --days-back 30

Then generate reports:

python3 US1_RPT_create_report_info_files.py --settings settings

How do I debug regex pattern matching?

Test in Python:

python3 -c "
import re
text = '[PROJ-123] Add feature'
pattern = r'PROJ-[0-9]+'
match = re.search(pattern, text)
print(f'Match: {match.group()}' if match else 'No match')
"

---

Best Practices

How should I organize my settings files?

Recommended structure:

base_settings          # Non-sensitive defaults
team_settings          # Team-specific config
credentials            # Tokens (gitignored)

Use multiple files:

python3 script.py --settings base_settings --settings credentials

Should I commit settings to git?

• Commit: Structure, regex patterns, test stages, non-sensitive URLs

• Don't commit: API tokens, passwords, production credentials

Add to .gitignore:

settings
credentials
*_credentials.json

How often should I generate reports?

Automated:

• Nightly for active sprints

• After each deployment

On-demand:

• Before sprint planning/review

• Before releases

• When coverage questions arise

What's the recommended lookback period?

• Tagging: 30-90 days (captures most active development)

• Coverage: 14-30 days (balances data freshness vs performance)

Adjust based on your development velocity.

---

Advanced Questions

Can I customize report formatting?

HTML/Confluence reports use embedded templates in the scripts. To customize, modify the Python scripts directly (search for HTML template strings).

Can I export data to other tools?

Yes! The ReportInfo JSON files contain structured data you can parse and send anywhere:

# Example: Send to custom endpoint
for report in reports/ReportInfo_*.json; do
  curl -X POST https://your-api.com/coverage \
    -H "Content-Type: application/json" \
    -d @"$report"
done

How do I handle private repos with 2FA?

Use SSH keys or personal access tokens (PATs), not passwords. See Installation - Git Authentication.

Can I run this in Docker?

Yes:

FROM python:3.9-slim
WORKDIR /app
COPY . .
RUN pip install requests
CMD ["python3", "US1_RPT_create_report_info_files.py", "--settings", "settings"]

---

More Help

• Troubleshooting Guide - Detailed error solutions

• Configuration Guide - All settings explained

• Examples - Working code samples

• CLI Reference - Complete command documentation