Child pages
  • Indore 10 EAP3 (build 41843) Release Notes

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The Project Administrator role will no longer include the Agent Manager role for the new TeamCity installations. The existing installations will not be affected by this change, but the new permissions are added to Project Managers and it is possible to exclude the inherited agent manager role manually. 

Flaky Test Detection

TeamCity now supports flaky test detection. A flaky test is a test that is unstable (can exhibit both a passing and a failing result) with the same code.

Flaky test detection is based on the following heuristics:

1. High flip rate (Frequent test status changes). A flip in TeamCity is a test status change — either from OK to Failure, or vice versa. The Flip Rate is the ratio of such "flips" to the invocation count of a given test, measured per agent, per build configuration, or over a certain time period (7 days by default). A test which constantly fails, while having a 100% failure rate, will have its flip rate close to zero; a test which "flips" each time it is invoked will have the flip rate close to 100%.

If the flip rate is too high, TeamCity will consider the test flaky.

2. Different test status for build configurations with the same VCS change: if two builds of different configurations are run on the same VCS change and the test result is different in these builds, the test is reported as flaky. This may be an indication of environmental issues.

3. If the status of a test 'flipped' in the new build with no changes, i.e. a previously successful test failed in a build without changes or a previously failing test passed in a build without changes, TeamCity will consider the test flaky.


4. Different test status for multiple invocations in the same build: if the same test is invoked multiple times and the test status flips, TeamCity will consider the test flaky.


Such tests are displayed on the dedicated project tab, Flaky Tests,  along with the total number of test failures, the flip rate for the given test and reasons for qualifying the test as a flaky one.

You can also see if the test is flaky when viewing the expanded stacktrace for a failed test on the build results page.

As with any failed test, you can assign investigations for a flaky test (or multiple tests). For flaky tests the resolution method is automatically set to 'Manual'; otherwise the investigation will be automatically removed once the test is successful, which does not mean that the flaky test has been fixed.

 

New Create project/build configuration button

...