Build Language is an extensible build automation DSL for defining builds in a declarative way. Generated into Ant, it leverages Ant execution power while keeping your sources clean and free from clutter and irrelevant details. Organized as a stack of MPS languages with ANT at the bottom, it allows each part of your build procedure to be expressed at a different abstraction level. Building a complex artifact (like an MPS plug-in) could be specified in just one line of code, if you follow the language conventions, but, at the same time, nothing prevents you from diving deeper and customize the details like file management or manifest properties.
As with many build automation tools, project definition is the core of the script. Additionally, and unlike most of the other tools, Build Language gives you full control over the output directory layout. The expected build result is defined separately in the build script and not as a part of some (third-party) plugin.
Every build script is made up of three parts. The first is dependencies, something required that comes already built. Think of libraries or third-party languages, for example. Next is the project structure. It contains declarations of everything you have in your repository and what is going to be built, as well as the required build parameters. Note that declaring an item here does not trigger its build unless it is needed, i.e. referred to from the last part of the script - the output layout. The output could be as straightforward as a set of plain folders and copied files, or much more complex with zipped artifacts such as packaged plug-ins or MPS languages. For example, to build a jar file out of Java sources you need to declare a Java module in the project structure and the respective jar file with a reference to the module in the output layout.
Thanks to MPS, Build Language comes with concise textual notation and an excellent editing experience, including completion and on-the-fly validation. Extension languages (or plugins if we stick to the terminology of the other build tools) add additional abstractions on top of the language. In our experience, it is quite an easy process to create a new one compared to developing Maven or Gradle plugins.
See an example below of a build script, which builds a plugin for Intellij IDEA:
Let's look at it closely. The header of the script consists of general script information: the name of the script (Complex in the screenshot), the file it is generated into (build.xml) and the base directory of the script (in the screenshot one can see it is relative to the script location ../../ as well as full path).
The body of the script consists of the following sections:
Build Language provides several built-in plugins.
The Java plugin adds capability to compile and package java code. Source code is represented as java modules and java libraries.
Java module defines its content (source folders locations) and dependencies on other modules, libraries and jars. In content section java module can have:
In dependencies section java module can have:
Each java module is generated into its own ant target depending on each other according to source module dependencies. For compiling cyclyc module dependencies two steps compilation is performed:
Java library consists of jars (either specified by path or as references to the other project layout) and class folders. The available elements are:
Compilation settings for java modules are specified in java options. There can be several java options in the build script, only one of them can be default. Each module can specify its own java options to be used for compilation.
Java plugin adds the following targets:
The MPS plugin enables the build language in scripts to build mps modules. In order to use the MPS plugin one must add jetbrains.mps.build.mps language into used languages.
The MPS plugin enables adding modules into project structure. On the screenshot there is an example of a language, declared in a build script.
Note that there is a lot of information about the module specified in the build script: uuid and fully qualified name, full path to descriptor file, dependencies, runtime (for a language) etc. This information is required for packaging the module. So, every time something changes for this module, for example a dependency is added, the build script has to be changed as well. There is of course a number of tools to do it easily. The typical process of writing and managing mps modules in the script looks as following:
Another thing to remember about MPS module declarations in a build scripts is that they do not rely on modules being loaded in MPS. All the information is taken from a module descriptor file on disk, while module itself could be unavailable from the build script.
MPS modules can be added into an mps group in order to structure the build script. An MPS Group is just a named set of modules, which can be referenced from the outside, for example one can add a module group into an IDEA plugin as one unit.
As it was written above, a lot of information about a module is extracted into build script and stored in it. This required constantly updating a script when module dependencies change. For a solution, it's both reexported and non-reexported dependencies are extracted to the script. For a language, apart from the dependencies, runtime solutions and extended languages are extracted.
"Building" a module with build script consists of two parts: generating this module and compiling module sources. Generating is an optional step for projects that have their source code stored in version control system and keep it in sync with their models. For generating, one target is created that generates all modules in the build script. Modules are separated into "chunks" – groups of modules that can be generated together – and generate task generates chunks one by one. For example, a language and a solution written on it can not be generated together, therefore they go into separate chunks. Apart from the list of chunks to generate, generate task is provided with a list of idea plugins to load and a list of modules from other build scripts that are required for generation. This lists of plugins and modules is calculated from dependencies and therefore their correctness is crucially important for successful generation. This is a major difference between generating a module from MPS and from a build script: while generating a module from MPS, generator has all modules in project loaded and available; however while generating a module from build script, generator only has whatever was explicitly specified in module dependencies. So build script can be used as a some kind of verifier of correctness of modules dependencies. Compilation of a module is performed a bit differently: for every MPS module a java module is generated, so in the end each MPS module is compiled by an ordinary ant javac task (or similar other task, if it was selected in java options).
So in order to generate and compile, dependencies of a module are to be collected and embedded into generated build xml file. Used languages and devkits are collected during generation of a build script from module descriptor files. The other information is stored inside build script node. On the screenshot below a module structure is shown for a project called "myproject" which uses some third-party MPS library called "mylibrary".
Arrows illustrate dependency system between modules. Purple arrows denote dependencies that are extracted to build script, blue arrows – dependencies that are not extracted. It can be easily seen that in order to compile and generate modules from myproject knowledge of "blue arrows" inside of mylibrary is not required. Which means that actual module files from my library may not even be present during myproject build script generation. Every information that generator needs is contained in the build script. Which is really very convenient: there is no need to download a whole library and specify its full location during build generation and generation process saves time and memory by not loading all module descriptors from project dependencies.
When an MPS solution contains test models, i.e models with the stereotype "@tests", they are generated into a folder "tests_gen" which is not compiled by default. To compile tests, one needs to specify in the build script that a solution has test models. This is done manually in the inspector. There are three options available for a solution: "with sources" (the default), "with tests" and "with sources and tests".
mps settings allow to change MPS-specific parameters for a build script. No more than one instance of mps settings can exist in the build script in the "additional aspects" section. Parameters that can be changed:
Projects that keep their generated source files in version control can check that these generated files are up-to-date using build script. After setting test generration in mps settings to true a call of gentest task appears in test target of generated build script. Similarly to generate task, gentest loads modules in the script, their dependencies from other build scripts and idea plugins that are required. For each module gentest task invokes two tests: "%MODULE_NAME%.Test.Generating" and "%MODULE_NAME%.Test.Diffing". Test.Generating fails when module has errors during generation and Test.Diffing fails when generated files are different from the ones on disk (checked out from version control). Test results and statistic are formatted into an xml file supported by TeamCity build server.
idea plugin construction defines a plugin for IntelliJ IDEA or MPS with MPS modules in it. In the screenshot you can see an example of such plugin.
The first section of the plugin declaration consists of various information describing the plugin: its name and description, the name of folder, the plugin vendor, etc.. The important string here is plugin id, which goes after the keywords idea plugin. This is the unique identifier of the plugin among all the others (in the example the plugin id is jetbrains.mps.samples.complex).
The next section is the actual plugin content – a set of modules or module groups included into the plugin. If some module included in the plugin needs to be packaged in some special way other than the default, this should also be specified here (see the line "custom packaging for jetbrains.mps.samples.complex.library").
The last section is dedicated to the plugin dependencies on other plugins. The rule is that if we have a "moduleA" located in plugin "pluginA", which depends on "moduleB" located in "pluginB", then there should be a dependency of "pluginA" on "pluginB". A typesystem check exists that will identify and report such problems.
The layout of the plugin is specified last:
In the screenshot, module jetbrains.mps.samples.complex.library is packaged into the plugin manually as it is specified in idea plugin construction not to package it automatically.
The MPS plugin provides the following targets:
The Module testing plugin, provided by jetbrains.mps.build.mps.tests language, adds to build scripts the capability to execute NodeTestCases in the MPS solutions. Tests are executed after all modules are compiled and packaged into a distribution, i.e. against the packaged code, so they are invoked in an environment close to the real use.
Solutions/module groups with tests are grouped into test modules configurations, which is a group of solutions with tests to be executed together in the same environment. All required dependencies (i.e. modules and plugins) are loaded into that environment.
In the screenshot, you can see a test modules configuration, named execution, which contains a solution jetbrains.mps.execution.impl.tests and a module group debugger-tests.
There is a precondition for solutions to be included into test a modules configuration. A solution should be specified as containing tests (by selecting "with tests" or "with sources and tests" in inspector). A module group should contain at least one module with tests.
Test results and statistic are formatted into an xml file (which is supported by TeamCity).
The following articles explain how to build a language plugin:
Articles on the topic of building with MPS: