Running tests

If .NetCore FW is not installed on your testing machine(s), you can download and execute the “Self-contained” version of the Sealights agent. See Downloading the .NET Core agent for details. In this case, replace in the commands below the beginning dotnet .\sl-dotnet-agent\SL.DotNet.dll with .\sl-dotnet-agent\SL.DotNet.exe for example (without dotnet and with the agent executable directly)

Running tests with MSTest, NUnit, or XUnit

You have two approaches to capture tests from a compatible framework: wrapping the command or using the Profiler-Initiated-Collector configuration. With the later approach, you don’t have to modify your startup command.

Minimum System Requirements

• MSTest Framework 1.4.0 (a.k.a. MSTest v2) and higher

• NUnit 3.10.0 and higher

• XUnit 2.3.0 and higher

• VSTest.Console.exe 17.2 and higher

Wrapper to the test command

When using MSTest, NUnit, or XUnit, you must wrap the execution with the agent and the test listener will monitor the tests. When test execution is completed, test results will be uploaded automatically.

Here is a sample command with the LabID-only setup:

Lab ID Only

dotnet ./sl-dotnet-agent/SL.DotNet.dll run --tags mytag --instrumentationMode tests --labId <lab Id> --testStage "Functional Tests" --target "c:\tools\testrunner.exe" --workingDir $(pwd) --targetArgs "MyTest.dll --flag1 true --flag2 false"

BuildSession ID

dotnet ./sl-dotnet-agent/SL.DotNet.dll run --tags mytag --instrumentationMode tests --buildSessionIdFile buildSessionId.txt --testStage "Functional Tests" --target "c:\tools\testrunner.exe" --workingDir $(pwd) --targetArgs "MyTest.dll --flag1 true --flag2 false"

For “In-Process Testing” (like Unit Tests, Integration Tests and Components Tests), you will need to remove the flag --instrumentationMode tests from the command above as the agent needs to capture coverage as well in this use case. Otherwise, you can provide an explicit value of testsAndCoverage.

See Command Reference for full parameter details

PIC Integration

Set environment variables required for profiling and PIC. In the below sample configuration, we consider that agent binaries are unpacked to the $SL_AGENT_ROOT, and the $SL_AGENT_ROOT variable contains an absolute path (relative path could cause issues). Env variables could be set by sourcing script, by setting env variables for docker, using env file for docker/ docker-compose / k8s, etc…

• Environment variables should be set in the same session where test runner will be executed.

• Avoid setting PIC or Profiling environment variables system-wide or user-wide. It could lead to unpredictable consequences because the profiler and collector will be started for each .NET process in the system. Also, it will conflict with other dotnet agents running in the same scope.

Windows

# enable .net profiler
Env:COR_ENABLE_PROFILING=1
Env:COR_PROFILER="{01CA2C22-DC03-4FF5-8350-59E32A3536BA}"
Env:COR_PROFILER_PATH_32="$SL_AGENT_ROOT\SL.DotNet.ProfilerLib.Windows_x86.dll"
Env:COR_PROFILER_PATH_64="$SL_AGENT_ROOT\SL.DotNet.ProfilerLib.Windows_x64.dll"
Env:CORECLR_ENABLE_PROFILING=1
Env:CORECLR_PROFILER="{01CA2C22-DC03-4FF5-8350-59E32A3536BA}"
Env:CORECLR_PROFILER_PATH_32="$SL_AGENT_ROOT\SL.DotNet.ProfilerLib.Windows_x86.dll"
Env:CORECLR_PROFILER_PATH_64="$SL_AGENT_ROOT\SL.DotNet.ProfilerLib.Windows_x64.dll"
# configure Sealights PIC params. 
Env:SL_PROFILER_INITIALIZECOLLECTOR=1
Env:SL_PROFILER_INSTRUMENTATIONMODE="tests"
#Env:SL_COVERAGECOLLECTOR_STARTUPINACTIVITYTIMEOUTSEC=3
#Env:SL_COVERAGECOLLECTOR_IDLETIMEOUTSEC=3
# configure env specific params (token, bsid, labid...)
Env:SL_SESSION_TOKENFILE="$SL_AGENT_ROOT\sltoken.txt"
Env:SL_SESSION_BUILDSESSIONIDFILE="$SL_AGENT_ROOT\buildSessionId"
Env:SL_GENERAL_TESTSTAGE="Functional Tests"
#Env:SL_LABID="integ_poc_app"

Linux

#!/bin/bash
# enable .net profiler
export CORECLR_ENABLE_PROFILING=1
export CORECLR_PROFILER="{3B1DAA64-89D4-4999-ABF4-6A979B650B7D}""
export CORECLR_PROFILER_PATH_64="$SL_AGENT_ROOT/libSL.DotNet.ProfilerLib.Linux.so"
export CORECLR_PROFILER_PATH_32="$SL_AGENT_ROOT/libSL.DotNet.ProfilerLib.Linux.so"
# configure Sealights PIC params. 
export SL_PROFILER_INITIALIZECOLLECTOR=1
export SL_PROFILER_INSTRUMENTATIONMODE="tests"
#export SL_COVERAGECOLLECTOR_STARTUPINACTIVITYTIMEOUTSEC=3
#export SL_COVERAGECOLLECTOR_IDLETIMEOUTSEC=3
# configure env specific params (token, bsid, labid...)
export SL_SESSION_TOKENFILE="$SL_AGENT_ROOT/sltoken.txt"
export SL_SESSION_BUILDSESSIONIDFILE="$SL_AGENT_ROOT/buildSessionId"
export SL_GENERAL_TESTSTAGE="Functional Tests"
#export SL_LABID="integ_poc_app"

You can now execute your tests in the same session, using regular commands with any needed set of params:

dotnet test ./SomeAPITests.dll --params /settings -flags -cookies -filters -kittens

In the environment variables set, SL_PROFILER_INCLUDEPROCESSFILTER is not specified on purpose. If you run test using dotnet test SomeAPITests.dll, usually there will be executed up to three subprocesses (depending on Test FW and OS):

1. dotnet test SomeAPITests.dll - as a host process, doesn’t execute any tests, just manage child processes, parse config, wrap different test runners

2. dotnet exec "TestFrameworkTestRunner executable" SomeAPITests.dll --adjusted flags - listen over IPC messages from real test executor + test executer host

3. dotnet --runtimeconfig Path\to\generated\runtimeconfig.json --some --generated -flags - real test executor, which loads testdlls and produces test events. We need to be attached to this process

So if you specify in the filter only dll, we could attach to the wrong process. Therefore, best practice is to avoid declaring the filter and set in the same session where test runner is executed.

Legacy approach

See Command Reference for full parameter details

Starting a test stage

Before running your tests, you must send the SeaLights server a start execution event with the test stage's name.

dotnet ./sl-dotnet-agent/SL.DotNet.dll startExecution --buildSessionIdFile buildSessionId.txt --testStage "Functional Tests"

Running your tests

Now you can run your tests while capturing coverage: Capturing coverage from the app process.

If you’re using a non-supported testing framework, execute your tests with your regular command/script and once the tests are finished, you can upload one or more report files. Supported formats are MSTest (TRX files), xUnit.Net, and NUnit (XML files)

dotnet ./sl-dotnet-agent/SL.DotNet.dll uploadTestReport --buildSessionIdFile buildSessionId.txt --testStage "Functional Tests" --report myReport.trx

Ending the test stage

Once you run your tests and upload the results, you must send the SeaLights server an end execution event with the test stage's name.

dotnet ./sl-dotnet-agent/SL.DotNet.dll endExecution --buildSessionIdFile buildSessionId.txt --testStage "Functional Tests"

Sample scripts

NUnit Tests integration via PIC

Windows

# enable .net profiler
$Env:COR_ENABLE_PROFILING=1
$Env:COR_PROFILER="{01CA2C22-DC03-4FF5-8350-59E32A3536BA}"
$Env:COR_PROFILER_PATH_32="C:\Sealights\agent\SL.DotNet.ProfilerLib.Windows_x86.dll"
$Env:COR_PROFILER_PATH_64="C:\Sealights\agent\SL.DotNet.ProfilerLib.Windows_x64.dll"
$Env:CORECLR_ENABLE_PROFILING=1
$Env:CORECLR_PROFILER="{01CA2C22-DC03-4FF5-8350-59E32A3536BA}"
$Env:CORECLR_PROFILER_PATH_32="C:\Sealights\agent\SL.DotNet.ProfilerLib.Windows_x86.dll"
$Env:CORECLR_PROFILER_PATH_64="C:\Sealights\agent\SL.DotNet.ProfilerLib.Windows_x64.dll"
# configure Sealights PIC params.
$Env:SL_PROFILER_INITIALIZECOLLECTOR=1
$Env:SL_PROFILER_INSTRUMENTATIONMODE="tests"
$Env:SL_COVERAGECOLLECTOR_STARTUPINACTIVITYTIMEOUTSEC=3
$Env:SL_COVERAGECOLLECTOR_IDLETIMEOUTSEC=3
# configure env specific params (token, bsid, labid...)
$Env:SL_SESSION_TOKENFILE="C:\Sealights\agent\sltoken.txt"
$Env:SL_LABID="TestLab"
$Env:SL_GENERAL_TESTSTAGE="Functional Tests"
# Logging 
$Env:SL_LOGGING_ENABLED="true"
$Env:SL_LOGLEVEL=6
$Env:SL_LOGGING_TOFILE="true"
$Env:SL_LOGDIR="C:\Sealights\logs\Functional"
$Env:SL_LOGGING_FILENAME="C:\Sealights\logs\Functional\pic.log"

#Execute your regular testing command

NUnit Tests integration with command wrapper

Windows

# Download the SL DotNetCore Agent
$global:ProgressPreference = "SilentlyContinue"
iwr -OutFile sealights-dotnet-core-agent.zip -UseBasicParsing -Uri https://agents.sealights.co/dotnetcore/latest/sealights-dotnet-agent-windows-self-contained.zip
Expand-Archive .\sealights-dotnet-core-agent.zip -DestinationPath sl-dotnet-agent -Force
Write-Output "[Sealights] .NetCore Agent version is: $(Get-Content .\sl-dotnet-agent\version.txt)"

# Place the token in the agent folder
Copy-Item "C:\Sealights\sltoken.txt" -Destination ".\sl-dotnet-agent\"
#Out-File -InputObject $env:SEALIGHTS_AGENT_TOKEN -NoNewline -Force -FilePath ".\sl-dotnet-agent\sltoken.txt"

$env:SL_LABID = "my_labID"
$env:SL_TESTSTAGE_NAME = "Functional Tests"

# Setup Sealights log level
$Env:SL_LogDir = ".\sl-logs\"
$Env:SL_LogLevel = "6"
$Env:SeaLights_EnableInjectedLibLogFile = "1"

# Pass the NUnit Command to SL Proceess Wrapper using Predefined Agent's Env variables to avoid challenges with quotes, file paths and other variables interpetation
# Prefer using only double quotes " and escape quotes ` when passing a complex command line to the SL Agent (Powershell)
$Env:SL_PROFILER_TARGET="C:\QATools\.nuget\packages\nunit.consolerunner\3.10.0\tools\nunit3-console.exe"
$Env:SL_PROFILER_TARGETARGS="`"C:\AutomationTests\ui-e2e-tests\AutomationTests\Tests\bin\Debug\UI.E2E.Tests.dll`" --inprocess --where cat=E2E_P1 --params=APP_OS=Linux;CollectLogs=`"True`" "

# Execute the Tests using SL Agent as a wrapper to the NUnit process
.\sl-dotnet-agent\SL.DotNet.exe run --tags nunit --instrumentationMode tests --labId $env:SL_LABID --testStage $env:SL_TESTSTAGE_NAME --workingDir $(pwd) --enablePerformanceMonitoring true