Unable to render embedded object: File (TeamCity48.png) not found.

TeamCity 10.x and 2017.x Documentation

Icon

You are viewing the documentation of TeamCity 10.x and 2017.x, which is not the most recently released version of TeamCity.
View this page in the latest documentation or refer to the listing to choose the documentation corresponding to your TeamCity version.

 

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: updated the flaky failure heuristics

...

  1. High flip rate (Frequent test status changes). A flip in TeamCity is a test status change — either from OK to Failure, or vice versa. The Flip Rate is the ratio of such "flips" to the invocation count of a given test, measured per agent, per build configuration, or over a certain time period (7 days by default). A test which constantly fails, while having a 100% failure rate, will have its flip rate close to zero; a test which "flips" each time it is invoked will have the flip rate close to 100%.
    If the flip rate is too high, TeamCity will consider the test flaky.
  2.  Different Different test status for build configurations with the same VCS change: if two builds of different configurations are run on the same VCS change and the test result is different in these builds, the test is reported as flaky. This may be an indication of environmental issues.
  3. If the status of a test 'flipped' in the new build with no changes, i.e. a previously successful test failed in a build without changes or a previously failing test passed in a build without changes, TeamCity will consider the test flaky.
  4. Different test status for multiple invocations in the same build (flaky failure): if the same test is invoked multiple times and the test status flips, TeamCity will consider the test flaky.
    This heuristic is supported for TestNG unit tests with  invocationCount [1][2] greater than 1 . Additionally, for JUnit tests, a 3rd party tempus-fugit library can be used together with JUnit. It is sufficient to annotate a test with  @Intermittent  and use the  IntermittentTestRunner  test runner, as in the minimal example below:

    Code Block
    import org.junit.Test;
    import org.junit.runner.RunWith;
    
    import com.google.code.tempusfugit.concurrency.IntermittentTestRunner;
    import com.google.code.tempusfugit.concurrency.annotations.Intermittent;
    
    @RunWith(IntermittentTestRunner.class)
    public class MultipleViaTempusFugit {
        @Test
        @Intermittent(repetition = 10)
        public void test() {
            // ...
        }
    }

    If such a test flakes at least once at multiple invocations within a single build, the Flaky Failure heuristic will be triggered.


Such tests are displayed on the dedicated project tab, Flaky Tests,  along with the total number of test failures, the flip rate for the given test and reasons for qualifying the test as a flaky one.

...