FAQ
Find common answers about setup, tagging, reporting, and integrations.
Common answers about setup, tagging, reporting, and integrations.
Use Troubleshooting when a run fails, returns no data, or produces the wrong result.
General Questions
What is this tool for?
User Story Coverage connects issue tracking data and SeaLights coverage data.
It tags code changes with ticket IDs and generates coverage reports at the user story level.
This helps teams see which features have enough test coverage.
Do I need to install anything on my servers?
No.
This is a Python command-line tool that runs wherever Python 3.7 or later is available.
It connects to your existing systems through their APIs.
Which platforms are supported?
Issue trackers
Jira Cloud
Jira on-premises with plugin support
Azure DevOps
Source control
GitHub
GitLab
Bitbucket
Azure DevOps Git
Coverage platform
SeaLights
How much does it cost?
The tool is included as part of the SeaLights integration.
You still need active access to SeaLights, your issue tracker, and your source control platform.
Setup and Configuration
How do I get started?
Follow the shortest working path below.
Install and run once
Complete Installation and first run.
Fill in your settings
Use Configuration for the full settings model.
Pick a workflow
Continue with the matching path in Deployment patterns.
Where do I get API tokens?
Get tokens from each source system:
SeaLights — create an API token in Settings → API Tokens
Jira Cloud — create an Atlassian API token
GitHub — create a personal access token with repository read access
Azure DevOps — create a PAT with the required work item permissions
Use Credential Management for storage, rotation, and minimum-permission guidance.
Can I use environment variables instead of a settings file?
Yes.
All settings can be overridden with environment variables.
Use Environment variables for the full variable list and examples.
Can I use multiple settings files?
Yes.
You can pass multiple --settings values and merge configurations from left to right.
Use Configuration for merge rules and examples.
Tagging Questions
How does the tool extract ticket IDs?
The tool uses a regex pattern to extract ticket IDs from commit messages, branch names, and pull request titles.
Configure the pattern in your SCM settings with key_pattern_regex.
Do I need to tag code before generating reports?
It depends on your tagging mode.
If SeaLights auto-tagging is enabled, you can skip the source-tagging scripts.
If auto-tagging is not enabled, run the tagging phase before the reporting phase.
Use Source Tagging for the tagging commands and Installation and first run for the shortest working path.
What is SeaLights auto-tagging and how do I enable it?
SeaLights auto-tagging extracts ticket IDs from git commit data during build reporting.
It removes the need to run the US_SRC_* tagging scripts.
To enable it, contact your SeaLights Customer Success representative and provide your ticket ID regex pattern plus the app and branch scope.
How often should I run tagging?
It depends on your workflow:
Daily or continuous — run after merges or on a nightly schedule
Sprint-based — run at sprint start, mid-sprint, and sprint end
On-demand — run before releases or for a specific report request
If SeaLights auto-tagging is enabled, you do not need a separate tagging schedule.
Can I tag retroactively?
Yes.
Use the --since-date option with the GitHub PR tagging flow when you need historical tagging.
Use Source Tagging for the exact command and supported tagging paths.
What if I don't use pull requests?
Use the git history tagging flow instead of the PR-based flow.
This works well when your team merges directly to branches or does not rely on pull requests.
Use Source Tagging for the history-based command and --days-back usage.
Reporting Questions
What formats are reports available in?
Reports are available as:
JSON data files
HTML pages
Confluence pages
Jira or Azure DevOps field updates
Can I customize the HTML reports?
Not through standard settings.
HTML output is generated from templates embedded in the scripts.
How long does report generation take?
Runtime depends on ticket count, app and branch scope, SeaLights API response time, and coverage_days_back.
A typical run takes 5 to 15 minutes for 20 to 50 tickets.
Can I speed up report generation?
Yes.
Start with these changes:
reduce
coverage_days_backnarrow your JQL or WIQL query
reuse an existing report with
--report-idprocess tickets in smaller batches
What if a ticket has no coverage data?
The report shows 0% coverage.
This usually means the code was not tagged with that ticket ID, no tests ran against the tagged code, or the lookback period missed the relevant test runs.
Can I customize report formatting?
Standard settings do not expose full report templating.
HTML and Confluence output use embedded templates in the scripts.
Check Configuration before changing templates directly.
Can I export data to other tools?
Yes.
ReportInfo_*.json files contain structured data that you can parse and send to other systems for dashboards, notifications, and gates.
Use Report Generation for the report outputs and Deployment patterns for automation patterns.
Ticket Management Questions
What JQL should I use?
It depends on your reporting goal.
Current sprint
Last 30 days
Specific epic
By labels
Use Data sources for the full Jira query settings model.
How do I find custom field IDs?
Inspect a Jira issue through the Jira REST API and locate the matching customfield_XXXXX entry in the response.
Use Publication targets for legacy Jira publishing behavior and field-mapping context.
Do I need admin access to Jira?
No, but you do need:
read access to projects in your JQL
write access if you update custom fields through Jira publishing
plugin admin rights if you use the Jira plugin
Some Jira environments require elevated credentials when updates are sent with notifyUsers=false. If your token cannot do that, set jira.notify_users or JIRA_NOTIFY_USERS to true.
What WIQL should I use?
Choose a query that matches your work item scope.
User stories in last 30 days
Specific iteration
By area path
Use Data sources for the full Azure DevOps query settings model.
What permissions do I need in Azure DevOps?
Create a PAT with these permissions:
Work Items — Read
Extension Data — Read and Write, if you use the Azure DevOps plugin
CI/CD Questions
Can I run this in CI/CD?
Yes. Deployment patterns has runnable pipeline configurations for GitHub Actions, Jenkins, GitLab CI, and Azure Pipelines.
Can I run reports on a schedule?
Yes.
Run the reporting flow from your CI/CD scheduler or a cron-based trigger.
Can I fail builds based on coverage thresholds?
Yes.
Parse the generated JSON output and apply your own threshold gate in the pipeline.
Use Report Generation for the output files and Deployment patterns for workflow patterns.
What outputs should I publish as artifacts?
Publish the outputs that match your pipeline goal:
HTML reports for review
ReportInfo_*.jsonfiles for automation and gatingconfluence_links.jsonif you publish Confluence pages
Can I send notifications from the pipeline?
Yes.
The JSON outputs can feed Slack, Microsoft Teams, email, or custom webhook notifications.
Can I export report data to other tools?
Yes.
Use the generated JSON files for downstream automation, dashboards, and notifications.
Do I need a SeaLights agent in the reporting job?
No.
The reporting tool reads coverage through the SeaLights REST API.
You still need SeaLights agents running during tests and builds reported to SeaLights with the correct app and branch names.
Repository Mapping Questions
Does it work with monorepos?
Yes.
Configure multiple entries in repo_list, either for different paths or for the same repository with different filters.
Can I track coverage across multiple repositories?
Yes.
List all repositories in repo_list and map each one to the matching SeaLights application.
What if my app name in SeaLights is different from my repo name?
That is fine.
Map the repository name to the SeaLights application name in repo_list.
What are integration builds?
Integration builds combine coverage from multiple component apps.
Enable them in coverage_app_list and make sure each component also exists in repo_list.
Can I use regex for app and branch names?
Yes.
Use app_regex and branch_regex in coverage_app_list to match multiple applications or branches.
Test Stage Questions
What are test stages?
Test stages represent phases of testing such as unit, integration, and end-to-end.
They must match the stage names configured in SeaLights.
How do I know what my test stages are called?
Check your SeaLights dashboard or Test Gap Analysis reports.
Use the same names in your settings.
What are grouped test stages?
Grouped test stages aggregate coverage from multiple individual stages.
For example, one grouped stage can combine unit, integration, and end-to-end coverage into a single rollup.
Use grouped_test_stages for this configuration.
Can I add custom test stages?
Yes, as long as they already exist in SeaLights.
Add them to the test_stages array in your settings.
Performance Questions
Why is tagging so slow?
Common causes include a large time range, many pull requests or commits, large repositories, and network latency to your SCM platform.
Start by reducing --days-back, limiting repositories per run, and running closer to your SCM platform when possible.
Why is report generation slow?
Common causes include many tickets in one run, long coverage_days_back windows, and first-run processing in SeaLights.
Start by reducing coverage_days_back, splitting large ticket sets into batches, and reusing an existing report with --report-id when possible.
Can I process repositories in parallel?
Yes.
You can run multiple tagging jobs at the same time when repositories are independent.
Use Deployment patterns for end-to-end workflow patterns.
Advanced Questions
Last updated
Was this helpful?