• [PyCharm IDE]
Skip to end of metadata
Go to start of metadata

PyCharm test runners protocol

General idea

Warning: java part of this API is for internal usage only. See Custom test runner example for real custom test runner example

There are a lot of differrent python test frameworks: pytest, nose, unittest to name some. Each framework supports plenty of plugins on top of it, so development/maintenance effort to support all of them may be huge. This document suggests how to extract test runners from PyCharm and declares clean and well-documented API that any runner should support. Here are major benefits: 

  • Test runners may be shared with TeamCity that also should be available to launch python tests, and they'd better run on TC in the same way they run in PyCharm
  • PyCharm-side of test support may be simplified dramatically since PyCharm may be agnostic about concrete test framework: it just provides functions, classes and modules, and accepts result tree.
  • There are alot of cases already covered by runners created by Leonid: https://github.com/JetBrains/teamcity-messages
  • Open and documented API actually provides plugin point giving users ability to support new runners or improve current ones by python-only coding. We may get some pull requests.

System overview

Pycharm launches dedicated python script called runner. Each runner should provide output in protocol, covered here: Build Script Interaction with TeamCity but with some extensions (see "generate tree" feature below) with aid of several modules: _jb_runner_tools.py that provides functions to start protcol, exception to be used when some location is not supported (see below), tools to parse options (ArgParse-based) of "what-to-run" subprotocol etc. messages.py implemenets protocol itself. Protocol works both for PyCharm and TC. So, to support new framework one must provide runner.

  • runner: accetps arguments from PyCharm, launches framework in framrwork-specific manner, returns data in TC protocol format.
  • framework: python framework itslef like pytest or nosetest
  • Runner utility functions: tools, used by runner to implement protocol

PyCharm to runner

PyCharm launches runner in the following format: runner_name.py [--target=..|–-path=..] -- custom_arguments

  • [–path=..|–target=..]: This is so-called "what-to-run" subprotocol. It is described below. Runner converts it to framework-specific format to launch appropriate tests. 
  • --: Separator. All arguments after it are passed to framework as-is, just like in sudo(8).
  • custom_arguments: framework-specific arguments. 

Runner builds command line for framework using "what-to-run" and "custom_arguments", reports it to user (runner must do that to make it transparent) and launches it. It depends on runner and framework if custom arguments go before or after "what-to-run".

"Target" subprotocol

When PyCharm wants runner to launch some test, it provides arguments using "target" subprotocol. There are several types of them:

  • Python target: it could be module, class or function (bare-function or class method). It is coded in python notation (dot separated) against one of PYTHONPATH folders. Provided as --target option.
    • Exmaples: Run everything in module in my_tests in package tests: "jb_nose_runner.py --target=tests.my_tests". Run test "test_test" in test case "MyTest": "jb_nose_runner.py --target=MyTest.test_test". Runner may support several targets. S
      • Since 2017.1.2 one may also provide path to file and name relative to this file using "::" as separator. For example: spam/eggs.py::MyTest.
  • Path target: Could be folder or file. Runner should use test framework to discover and run all tests in this folder. It is up to runner to do it recursively or not, but recursively is always better is possible. Folder provided as "–path=" option. Runner uses utility function to fetch it from command line options. Only one path is supported.
    • Example: _jb_unittest_runner.py --path=/path/to/my/tests
  • Raw arguments: User provides framework-specific arguments, and runner should pass them to framework as-is. This option is not recommended and should only be used by experts who want to launch something that is not supported by PyCharm.
    • Exmaple: _jb_unittest_runner.py -- discover -s /foo/bar/ -p "my_test*_spam*.py"

All three types are optional for runner. Some runners may support only some of them. In case when type is not supported, runner must raise WhatToRunNoSupportedException imported from utility functions. 

Any user-specific arguments are passed after "what-to-run" and should be obtained with utility functions.

Runner is encouraged to report arguments as they are provided to framework to help user with debugging issues.

Runner to PyCharm: how to report tests

Each test is named using Python naming scheme, although subtest extends this schema slightly:  module_name.ClassName.test_name.subtest_name. (subtest could be parametrized instance of tests which is used in some frameworksSome parts may be optional depending on context. Module name should be qualified against current PYTHONPATH.

 When test runners reports PyCharm test failure or success it should use as qualified name as possible. 

    • module_name
      • ClassName
        • test_name
        • test2_name
          • subtest_name
          • subtest2_name
        • test3_name
      • bare_function_test_like_in_pytest
    • module_name2..

PyCharm resolves test names in most cases, but not always: E.g. user provides folder external to sources. Navigation and "Rerun" does not work if names could not be resolved.


_jb_runner_tools "generate tree" feature

_jb_runner_tools splits tests to trees using several strategies, so at least one of them should be used by runner author. PyCharm assumes that each level must have "nodeId" and "parentNodeId" to build trees, but this functionaluty is handled by __jb_runner_tools, so runners should only provide flat tests to support following strategies:

  • Split-by-Dot: Test hierarchy is coded by imploding each level with dot. 
    • testStarted name="Parent.child"
    • testFinished name="Parent.child"
    • testStarted name="Parent.child2"
    • testFinished name="Parent.child2"
      and result is
    • Parent
      • child
      • child2
  • Block: You can use blockOpened/blockClosed messages from Build Script Interaction with TeamCity#blocksBlocksofServiceMessages . Each block starts new level.
  • Special: Some tests can't be splitted easily. Nose test-generators and py.test parametrized tests are good example. Yon need to parse their names to split them. Such strategy is runner specific, implemented in runner with aid of _jb_runner_tools


How PyCharm chooses runner

Several runners are hard-coded into PyCharm. How ever, user may choose custom runner and provide any runner what ever she likes. Runner should exist in PYTHONPATH and named in _jb_framrworkname_runner.py scheme.


  • No labels