Skip to end of metadata
Go to start of metadata

The Visual Studio Tests runner integrates MSTest runner and VSTest console runner. Support for both frameworks enables TeamCity to execute tests and automatically import their test results. 

The Visual Studio Test Runner requires Visual Studio Test Agent or Microsoft Visual Studio installed on the build agent.

 

On this page:

Visual Studio Tests runner settings

Option

Description

Test engine type

Select the tool used to run tests: VSTest or MSTest.

Test engine version

Select the version of the tool from the drop-down. By default, the available VSTest and MSTest installations are autodetected by TeamCity:

  • MSTest versions 2005-2015, and since TeamCity 10.0.4 - 2017 are supported.
  • VSTest versions 2012-2015, and since TeamCity 10.0.4 - 2017 are supported.

You can specify a custom path to the test runner here as well. TeamCity parameters are supported.

Icon

You can specify the agent property teamcity.dotnet.vstest.VS_VERSION.install.dir pointing to a VSTest distribution path and use it here, where VS_VERSION can be 11.0, 12.0,14.0. or since TeamCity 10.0.4 - 15.0. The property will have priority over registry values.

Test file names

This field is mandatory for VSTest and optional for MSTest.
Specify the new-line separated list of paths to assemblies to run tests on in the included assemblies list. Exclude assemblies from the test run by specifying paths to them in the corresponding field. Wildcards are supported.
(warning) Paths to the assemblies must be relative to the build checkout directory.

Run configuration file

(Optional) Specify the typical .runsettings file for VSTest or .testsettings file for MSTtest. You can use the file browser .

Additional command line parameters

Enter additional command line parameters for the selected test engine.
(info) Note that tests run with MSTest are not reported on-the-fly.
The Microsoft Developer Network lists the available options for VSTest and MSTest.

The rest of settings will vary depending on the engine to run tests with:

VSTest Settings

 

Option

Description

Target platform

Select the platform bitness. Note that specifying x64 target platform will force the vstest.console process to be run in the isolated mode

Framework

If the default is specified, vstest.console will select the target framework automatically. You can also choose the .Net platform manually using the drop-down.

Test names

(optional) Of all tests discovered in the included assemblies, only the tests with the names matching the provided values will be run. For multiple values, use new line.

If the field is empty, all tests will be run. See details in the Microsoft documentation.  (warning) Cannot be used with the option below.

Test case filter

Run tests that match the given expression. See details in the Microsoft documentation. (warning) Cannot be used with the option above.

Run in isolation

Run the tests in an isolated process

Use real-time test reporting

  • If this checkbox is selected, a custom TeamCity test logger will be used for real-time reporting. See the next section for details.
  • If not selected, VSTest will report tests to TeamCity after the run (the discovery of custom logger will not be attempted).

    Icon

    When using this option, it is recommended that you check the number of tests in the produced trx file against the build log. In case of incosistencies, switch to the custom logger.

     

Custom test logger

VSTest.Console supports custom loggers, i.e. libraries that can handle events that occur when tests are being executed.
TeamCity 9.0+ has a custom logger that provides real-time test reporting.
The logger must be installed manually on the agent machine, as it requires dlls to be copied to the Extensions folder of the VSTest.Console. No agent restart is needed when the custom logger is installed.

To install the custom logger:

    1. Download the custom logger
    2. Extract the contents of the downloaded archive on the agent machine:
      • for VisualStudio 2017 - to PROGRAM_FILES(x86)\Microsoft Visual Studio\2017\<Edition>\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensions
      • for VisualStudio 2015 - to PROGRAM_FILES\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensions
      • for VisualStudio 2013 - to PROGRAM_FILES\Microsoft Visual Studio 12.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensions
      • for VisualStudio 2012 - to PROGRAM_FILES\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensions
    • Check that the custom logger was installed correctly by executing vstest.console.exe /ListLoggers in the console on the agent machine. If the logger was installed correctly, you will see the logger with FriendlyName TeamCity listed:

MSTest settings

 

Option

Description

MSTest metadata

Enter a value for /testmetadata:file argument. See details in the Microsoft documentation

Testlist from metadata to run

Edit the Testlist. Every line will be translated into /testlist:line argument. See details in the Microsoft documentation. 

Test

Names of individual tests to run. This option will be translated into the series of /test: arguments. See details in the Microsoft documentation. 

Unique

Run the test only if one unique match is found for any specified test in test section

Results file

Enter a value for the /resultsfile:file name command-line argument. Parameter references are supported.
It is recommended to leave the field blank to generate a temporary *.trx file in the build temporary directory. Note that the directory may be cleaned between the builds.

To save the test run results to a named non-default file, enter a value for the /resultsfile:file name command-line argument.

    • If the path specified is relative, it will be relative to the build checkout directory. If the specified file already exists in the checkout directory, the build agent will attempt to delete the file. If the agent fails to delete it, the build will fail.

      Icon

      If you need to generate your results file in the checkout directory, consider adding the Swabra build feature to your configuration.

      To create a *.trx report in the build temporary directory, use %system.teamcity.build.tempDir%.

    • If the path specified is absolute, it will be used "as is".

 

 

 

 

 

  • No labels