Configuring flaky test detection – Support | JetBrains

Question

How to customize the behavior of flaky test detection in TeamCity?

Answer

Flaky test detection can be configured using internal properties. Most of them are global. The ones that can be overridden on the project or build configuration level are noted as such.


`teamcity.flakyTestDetector.timeFrame.days`	Controls the size of the look-back window (in days) used to detect flaky tests. The default value is `7` days. It should be possible to override it at the project or build configuration level using parameters.
`teamcity.flakyTestDetector.enabled.heuristics`	A comma-separated list of enabled heuristics. Use `*` or leave empty for all. Available heuristics include: `DifferentStatusAtSameVcsChangeHeuristic`, `FlakyFailureHeuristic`, `FlipRateTooHighHeuristic`, and `FlippedInBuildWithoutChangesHeuristic`.
`teamcity.flakyTestDetector.buildsCount`	Controls the number of recent build statuses examined per build configuration to find flaky tests, especially during server startup or background re-analysis. Default is `40`. It should be possible to override it at the project or build configuration level using parameters.
`teamcity.flakyTestDetector.projectExternalIds`	A comma-separated list of project external IDs to monitor. Use `*` for all projects.
`teamcity.flakyTestDetector.waitForBuildCompletion`	If `true`, test analysis is deferred until the build completes. If `false` (default), analysis starts as soon as an individual test finishes.
`teamcity.flakyTestDetector.cleanupOnBuildCompletion`	If `true`, outdated failures are cleaned up upon each build completion instead of only during server cleanup events.
`teamcity.flakyTestDetector.retainStatistics`	Whether test statistics are retained across server restarts. Default is `true`.
`teamcity.flakyTestDetector.maxChangesCacheSize`	Maximum size of the cache used for VCS changes calculation.
`teamcity.flakyTestDetector.maxQueueSize`	Maximum size of the queue for storing test info during the restoration procedure. Default is `20000`.
`teamcity.flakyTestDetector.testLimit`	The maximum number of tests the detector is allowed to keep track of globally.

Notes

teamcity.flakyTestDetector.timeFrame.days acts as a hard cutoff for how far back in time the detector will look for failures, while teamcity.flakyTestDetector.buildsCount acts as a limit on the number of builds processed per configuration. In a high-throughput project, if you set buildsCount to 40 but run 100 builds a day, the detector will only look back through the last ~10 hours of history, regardless of whether timeFrame.days is set to 7. To capture tests that flake infrequently over a week in a high-cadence project, you must increase buildsCount significantly.
Performance considerations: Keeping track of more builds and a longer time frame increases the memory footprint.