Using the Parallel Tests build feature with any build runner

Question

The documentation on the Parallel Tests build feature mentions the automatic parallelization of tests for the Maven, Gradle, IntelliJ IDEA Project, and .NET build runners. Is it possible to run tests in parallel with other build runners?

Answer

The Parallel Tests build feature may be used with other build runners. However, some additional setup is required. TeamCity makes this setup possible by providing special parameters. In general, custom parallelization of the tests works as follows:

1. TeamCity checks the duration of the tests and calculates which tests can be split between the batches. For TeamCity to be able to do that, it is necessary to inform TeamCity about the tests by reporting them to TeamCity. That can be done by either: • making your testing tool send service messages to stdout; • importing an XML with the test results. Once the tests are reported to TeamCity and can be found on the Tests tab of the build results page, proceed to the next step.
2. TeamCity will create a text file for each batch containing the names of the tests to exclude from the batch. This file can be referenced in a testing tool/script to exclude specific tests. TeamCity will run the testing tool/script on different agents and provide each agent with a unique file containing the names of tests to exclude from the current batch.

Implementation example

This example does not use any real testing tool, but rather aims to provide a general idea of how TeamCity can help parallelize tests run by any custom tool. For this example, the PowerShell build runner will be used. The test logic is irrelevant, so instead of running actual tests, the sleep command to simulate test execution will be used.

echo "##teamcity[testStarted name='MyTestSuite.Class1.TestX']"
sleep 60 # Pretend I'm testing something
echo "##teamcity[testFinished name='MyTestSuite.Class1.TestX' duration='60000']"

echo "##teamcity[testStarted name='MyTestSuite.Class2.TestY']"
sleep 60 # Pretend I'm testing something else
echo "##teamcity[testFinished name='MyTestSuite.Class2.TestY' duration='60000']"

After the Parallel Tests build feature is enabled, TeamCity will provide the system.teamcity.build.parallelTests.excludesFile parameter. The value of the parameter points to an automatically created file named excludedTests.txt. That file contains the names of the excluded from the current batch tests. In the case of two batches, the tests can be run in parallel on two agents. The content of the excludedTests.txt file for one batch is:

#version=1
#algorithm=duration
#current_batch=1
#total_batches=2
#creation_date=2022-08-18 15:38:28
#suite=
MyTestSuite.Class1

And for the other batch:

#version=1
#algorithm=duration
#current_batch=2
#total_batches=2
#creation_date=2022-08-18 15:38:28
#suite=
MyTestSuite.Class2

The file on each agent excludes different classes. The first file excludes MyTestSuite.Class1, and the second one - MyTestSuite.Class2. Still, both agents run both tests, so there is no parallelization yet. That's because the test tool (i.e., PowerShell script) doesn't make use of the excludedTests.txt file. Therefore, it is necessary to make the test tool aware of it. This is where the system.teamcity.build.parallelTests.excludesFile parameter should be used. It can be used in the script to decide whether the test should run:

if ((cat %system.teamcity.build.parallelTests.excludesFile%) -notcontains "MyTestSuite.Class1") {
echo "##teamcity[testStarted name='MyTestSuite.Class1.TestX']"
sleep 60 # Pretend I'm testing something here
echo "##teamcity[testFinished name='MyTestSuite.Class1.TestX' duration='60000']"
}

if ((cat %system.teamcity.build.parallelTests.excludesFile%) -notcontains "MyTestSuite.Class2") {
echo "##teamcity[testStarted name='MyTestSuite.Class2.TestY']"
sleep 60 # Pretend I'm testing something else here
echo "##teamcity[testFinished name='MyTestSuite.Class2.TestY' duration='60000']"
}

As a result, each agent running will only run the test from its batch, but not the test from another batch, which splits the tests between the agents, allowing running the tests in parallel. Alternatively, the test tool/script can decide whether to run a test on the current agent by referring to the current batch number (and perhaps to the number of total batches) instead of reading the excludedTests.txt file. The parameters %teamcity.build.parallelTests.currentBatch% and %teamcity.build.parallelTests.totalBatches% may be used for such an approach. Example:

if (%teamcity.build.parallelTests.currentBatch% -eq 1) {
echo "##teamcity[testStarted name='MyTestSuite.Class1.TestX']"
sleep 60 # Pretend I'm testing something here
echo "##teamcity[testFinished name='MyTestSuite.Class1.TestX' duration='60000']"
}

if (%teamcity.build.parallelTests.currentBatch% -eq 2) {
echo "##teamcity[testStarted name='MyTestSuite.Class2.TestY']"
sleep 60 # Pretend I'm testing something else here
echo "##teamcity[testFinished name='MyTestSuite.Class2.TestY' duration='60000']"
}