Icon

You are viewing the documentation of TeamCity 10.x and 2017.x, which is not the most recently released version of TeamCity.
View this page in the latest documentation or refer to the listing to choose the documentation corresponding to your TeamCity version.

 

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The project overview page also displays the number of tests failed in build configurations as well as other problems. When the data in the problematic build configuration is not expanded, the link under the tests number takes you to the Current Problems page (see below).

When you expand the build configuration data, the problem summary appears. The presentation of problems is shortened if your browser does not have enough horizontal space.

...

  1. High flip rate (Frequent test status changes). A flip in TeamCity is a test status change — either from OK to Failure, or vice versa. The Flip Rate is the ratio of such "flips" to the invocation count of a given test, measured per agent, per build configuration, or over a certain time period (7 days by default). A test which constantly fails, while having a 100% failure rate, will have its flip rate close to zero; a test which "flips" each time it is invoked will have the flip rate close to 100%.
    If the flip rate is too high, TeamCity will consider the test flaky.
  2. Different test status for build configurations with the same VCS change: if two builds of different configurations are run on the same VCS change and the test result is different in these builds, the test is reported as flaky. This may be an indication of environmental issues.
  3. If the status of a test 'flipped' in the new build with no changes, i.e. a previously successful test failed in a build without changes or a previously failing test passed in a build without changes, TeamCity will consider the test flaky.
  4. Different test status for multiple invocations in the same build (flaky failure): if the same test is invoked multiple times and the test status flips, TeamCity will consider the test flaky.
    This heuristic is supported for TestNG unit tests with  invocationCount [1][2] greater than 1 . Additionally, for .

    Note

    There is a known issue with parameterized test invocations with TestNG: TeamCity relies on the Surefire and Failsafe Maven plugins for the purposes of test result reporting, and these plugins ignore test parameters in their XML output. As a result, parameterized test invocations within the same build are erroneously treated as multiple identical test invocations, and, if some of the invocations fail, the test is marked as flaky with the Flaky Failure reason. See the related issue.




    For
    JUnit tests, a 3rd party tempus-fugit library can be used together with JUnit. It is sufficient to annotate a test with  @Intermittent  and use the  IntermittentTestRunner  test runner, as in the minimal example below:

    Code Block
    import org.junit.Test;
    import org.junit.runner.RunWith;
    
    import com.google.code.tempusfugit.concurrency.IntermittentTestRunner;
    import com.google.code.tempusfugit.concurrency.annotations.Intermittent;
    
    @RunWith(IntermittentTestRunner.class)
    public class MultipleViaTempusFugit {
        @Test
        @Intermittent(repetition = 10)
        public void test() {
            // ...
        }
    }

    If such a test flakes at least once at multiple invocations within a single build, the Flaky Failure heuristic will be triggered.

...