What is MPS build language?
Build Language is an extensible build automation DSL for defining builds in a declarative way. Generated into Ant, it leverages Ant execution power while keeping your sources clean and free from clutter and irrelevant details. Organized as a stack of MPS languages with ANT at the bottom, it allows each part of your build procedure to be expressed at a different abstraction level. Building a complex artifact (like an MPS plug-in) could be specified in just one line of code, if you follow the language conventions, but, at the same time, nothing prevents you from diving deeper and customize the details like file management or manifest properties.
As with many build automation tools, project definition is the core of the script. Additionally, and unlike most of the other tools, Build Language gives you full control over the output directory layout. The expected build result is defined separately in the build script and not as a part of some (third-party) plugin.
Every build script is made up of three parts. The first is dependencies, something required that comes already built. Think of libraries or third-party languages, for example. Next is the project structure. It contains declarations of everything you have in your repository and what is going to be built, as well as the required build parameters. Note that declaring an item here does not trigger its build unless it is needed, i.e. referred to from the last part of the script - the output layout. The output could be as straightforward as a set of plain folders and copied files, or much more complex with zipped artifacts such as packaged plug-ins or MPS languages. For example, to build a jar file out of Java sources you need to declare a Java module in the project structure and the respective jar file with a reference to the module in the output layout.
Thanks to MPS, Build Language comes with concise textual notation and an excellent editing experience, including completion and on-the-fly validation. Extension languages (or plugins if we stick to the terminology of the other build tools) add additional abstractions on top of the language. In our experience, it is quite an easy process to create a new one compared to developing Maven or Gradle plugins.
Build script structure
See an example below of a build script, which builds a plugin for Intellij IDEA:
Let's look at it closely. The header of the script consists of general script information: the name of the script (Complex in the screenshot), the file it is generated into (build.xml) and the base directory of the script (in the screenshot one can see it is relative to the script location ../../ as well as full path).
The body of the script consists of the following sections:
- use plugins contains a list of plugins used in the script. Plugins in Build Language are similar to those in Gradle: they are extensions to the language that provide a number of tasks to do useful things, like compiling java code, running unit tests, packaging modules, etc. In the screenshot two plugins are used: java and mps, which means that the script can build java and mps code.
- macros section defines path macros and variables used in the project (idea_home and plugins_home) together with their default values, which could be overridden during execution of the script.
- dependencies defines script dependencies on other build scripts. If a script references to something defined in the other build script it must specify this script in the dependencies section. The example script on the screenshot depends on two other scripts IDEA and mpsPlugin. These are provided by MPS, so in order to use them one has to specify the artifacts location for them, i.e. place where ant can find the result of their work (in the example, idea_home should point to the location of Intellij IDEA jars and plugins_home should point to the location of MPS plugins for IDEA). One can as well depend on some build scripts in the same MPS project. In that case, artifacts location is not required and it is assumed that the required script would be built just prior to the current script (there is a target buildDependents to do so).
- project structure section contains the description of the project, i.e. which modules does it have, where the source code is located, what the modules classpath is etc. The example project in the screenshot consists of a single idea plugin named Complex and of a group of MPS modules.
- default layout defines how to package the project into the distribution. The example project on the screenshot is packaged into a zip file named Complex.zip.
- additional aspects defines some other things related to the project, for example, various settings, integration tests to run, and so on.
Build Language provides several built-in plugins.
The Java plugin adds capability to compile and package java code. Source code is represented as java modules and java libraries.
Java module defines its content (source folders locations) and dependencies on other modules, libraries and jars. In content section java module can have:
- folder – a path to source folder on disk;
- resources – a fileset of resources. Consists of a path to resources folder and a list of selectors (include, exclude or includes).
- content root – a root with several content folders.
In dependencies section java module can have:
- classpath – an arbitrary xml with classpath;
- external jar – a jar file from other build script layout;
- external jar in folder – a jar file referenced by name in a folder from some other build script layout;
- jar – a path to local jar;
- library – a reference to a java library;
- module – a reference to a java module.
Each java module is generated into its own ant target depending on each other according to source module dependencies. For compiling cyclyc module dependencies two steps compilation is performed:
- A "cycle" target compiles all modules in the cycle together.
- Each module in the cycle is compiled with the result of compilation of "cycle" target in classpath.
Java library consists of jars (either specified by path or as references to the other project layout) and class folders. The available elements are:
- classes folder – a folder with classes;
- external jar – a jar file from other build script layout;
- external jars from – a collection of jars from a folder from some other buils script layout;
- jar – a path to local jar;
- jars – a path to local folder with jars and a list of selectors (include, exclude or includes).
Compilation settings for java modules are specified in java options. There can be several java options in the build script, only one of them can be default. Each module can specify its own java options to be used for compilation.
Java plugin adds the following targets:
- compileJava compiles all java modules in the project.
- processResources extension point for additional resource processing.
- classes does all compilation and resource processing in the project. It depends on targets compileJava, processResources.
- test extension point target for running unit tests.
- check does all testing and checking of project correctness. It depends on target test.
The MPS plugin enables the build language in scripts to build mps modules. In order to use the MPS plugin one must add jetbrains.mps.build.mps language into used languages.
MPS modules and groups
The MPS plugin enables adding modules into project structure. On the screenshot there is an example of a language, declared in a build script.
Note that there is a lot of information about the module specified in the build script: uuid and fully qualified name, full path to descriptor file, dependencies, runtime (for a language) etc. This information is required for packaging the module. So, every time something changes for this module, for example a dependency is added, the build script has to be changed as well. There is of course a number of tools to do it easily. The typical process of writing and managing mps modules in the script looks as following:
- Adding a module to the script. One specifies, which type of module to add (a solution, a language or a devkit) and the path to the module descriptor file. Then the intention "Load required information from file" can be used to read that file and fill the rest of the module specification automatically.
- Reflecting the changes made in the module. One can check a model with build scripts using Model checker to find whether it is consistent with the module files. Model checker will show all problems in the script and allow you to fix them using "Perform Quick Fixes" button. Instead of Model checker one can use the same "Load required information from file" intention to fix each module individually.
Another thing to remember about MPS module declarations in a build scripts is that they do not rely on modules being loaded in MPS. All the information is taken from a module descriptor file on disk, while module itself could be unavailable from the build script.
MPS modules can be added into an mps group in order to structure the build script. An MPS Group is just a named set of modules, which can be referenced from the outside, for example one can add a module group into an IDEA plugin as one unit.
How generating and compiling MPS modules actually works
As it was written above, a lot of information about a module is extracted into build script and stored in it. This required constantly updating a script when module dependencies change. For a solution, it's both reexported and non-reexported dependencies are extracted to the script. For a language, apart from the dependencies, runtime solutions and extended languages are extracted.
"Building" a module with build script consists of two parts: generating this module and compiling module sources. Generating is an optional step for projects that have their source code stored in version control system and keep it in sync with their models. For generating, one target is created that generates all modules in the build script. Modules are separated into "chunks" – groups of modules that can be generated together – and generate task generates chunks one by one. For example, a language and a solution written on it can not be generated together, therefore they go into separate chunks. Apart from the list of chunks to generate, generate task is provided with a list of idea plugins to load and a list of modules from other build scripts that are required for generation. This lists of plugins and modules is calculated from dependencies and therefore their correctness is crucially important for successful generation. This is a major difference between generating a module from MPS and from a build script: while generating a module from MPS, generator has all modules in project loaded and available; however while generating a module from build script, generator only has whatever was explicitly specified in module dependencies. So build script can be used as a some kind of verifier of correctness of modules dependencies. Compilation of a module is performed a bit differently: for every MPS module a java module is generated, so in the end each MPS module is compiled by an ordinary ant javac task (or similar other task, if it was selected in java options).
So in order to generate and compile, dependencies of a module are to be collected and embedded into generated build xml file. Used languages and devkits are collected during generation of a build script from module descriptor files. The other information is stored inside build script node. On the screenshot below a module structure is shown for a project called "myproject" which uses some third-party MPS library called "mylibrary".
Arrows illustrate dependency system between modules. Purple arrows denote dependencies that are extracted to build script, blue arrows – dependencies that are not extracted. It can be easily seen that in order to compile and generate modules from myproject knowledge of "blue arrows" inside of mylibrary is not required. Which means that actual module files from my library may not even be present during myproject build script generation. Every information that generator needs is contained in the build script. Which is really very convenient: there is no need to download a whole library and specify its full location during build generation and generation process saves time and memory by not loading all module descriptors from project dependencies.
Sources and tests
When an MPS solution contains test models, i.e models with the stereotype "@tests", they are generated into a folder "tests_gen" which is not compiled by default. To compile tests, one needs to specify in the build script that a solution has test models. This is done manually in the inspector. There are three options available for a solution: "with sources" (the default), "with tests" and "with sources and tests".
mps settings allow to change MPS-specific parameters for a build script. No more than one instance of mps settings can exist in the build script in the "additional aspects" section. Parameters that can be changed:
- bootstrap – setting this flag to "true" indicates that there are some bootstrapping dependencies between modules in the script. Normally the flag is set to false. See Removing bootstrapping dependency problems for details.
- test generation – if set to true, build script tests modules generation and difference between generated files and files on disk. Files can be excluded from diff in excludes section.
- generation max heap size in mb – maximum heap size for generation and generation testing.
Testing Modules Generation
Projects that keep their generated source files in version control can check that these generated files are up-to-date using build script. After setting test generration in mps settings to true a call of gentest task appears in test target of generated build script. Similarly to generate task, gentest loads modules in the script, their dependencies from other build scripts and idea plugins that are required. For each module gentest task invokes two tests: "%MODULE_NAME%.Test.Generating" and "%MODULE_NAME%.Test.Diffing". Test.Generating fails when module has errors during generation and Test.Diffing fails when generated files are different from the ones on disk (checked out from version control). Test results and statistic are formatted into an xml file supported by TeamCity build server.
idea plugin construction defines a plugin for IntelliJ IDEA or MPS with MPS modules in it. In the screenshot you can see an example of such plugin.
The first section of the plugin declaration consists of various information describing the plugin: its name and description, the name of folder, the plugin vendor, etc.. The important string here is plugin id, which goes after the keywords idea plugin. This is the unique identifier of the plugin among all the others (in the example the plugin id is jetbrains.mps.samples.complex).
The next section is the actual plugin content – a set of modules or module groups included into the plugin. If some module included in the plugin needs to be packaged in some special way other than the default, this should also be specified here (see the line "custom packaging for jetbrains.mps.samples.complex.library").
The last section is dedicated to the plugin dependencies on other plugins. The rule is that if we have a "moduleA" located in plugin "pluginA", which depends on "moduleB" located in "pluginB", then there should be a dependency of "pluginA" on "pluginB". A typesystem check exists that will identify and report such problems.
The layout of the plugin is specified last:
In the screenshot, module jetbrains.mps.samples.complex.library is packaged into the plugin manually as it is specified in idea plugin construction not to package it automatically.
The MPS plugin provides the following targets:
- generate - generates the mps modules that are included in the project structure.
- cleanSources - cleans the generated code (only for modules without bootstrapping dependencies). See more about bootstrapping dependencies in article "Removing bootstrapping dependency problems".
- declare-mps-tasks - a utility target that declares mps tasks such as generate or copyModels.
- makeDependents - invokes the generate target for a transient closure of this script's dependencies (if there is one) and then invokes assemble to put them together. It is guaranteed that each script is executed only after all its dependencies have been built.
Module Testing plugin
The Module testing plugin, provided by jetbrains.mps.build.mps.tests language, adds to build scripts the capability to execute NodeTestCases in the MPS solutions. Tests are executed after all modules are compiled and packaged into a distribution, i.e. against the packaged code, so they are invoked in an environment close to the real use.
Test modules configurations
Solutions/module groups with tests are grouped into test modules configurations, which is a group of solutions with tests to be executed together in the same environment. All required dependencies (i.e. modules and plugins) are loaded into that environment.
In the screenshot, you can see a test modules configuration, named execution, which contains a solution jetbrains.mps.execution.impl.tests and a module group debugger-tests.
There is a precondition for solutions to be included into test a modules configuration. A solution should be specified as containing tests (by selecting "with tests" or "with sources and tests" in inspector). A module group should contain at least one module with tests.
Test results and statistic are formatted into an xml file (which is supported by TeamCity).
The following articles explain how to build a language plugin:
Articles on the topic of building with MPS: