Skip to end of metadata
Go to start of metadata
Icon

For a quick how-to document on the MPS generator please check out the Generator Cookbook.

Introduction

Generator is a part of language specification that defines the denotational semantics for the concepts in the language.

MPS follows the model-to-model transformation approach. The MPS generator specifies translation of constructions encoded in the input language into constructions encoded in the output language. The process of model-to-model transformation may involve many intermediate models and ultimately results in sn output model, in which all constructions are in a language whose semantics are already defined elsewhere.

For instance, most concepts in baseLanguage (classes, methods etc) are "machine understandable", therefore baseLanguage is often used as the output language.

The target assets are created by applying model-to-text transformation, which must be supported by the output language. The language aspect to define model-to-text transformation is called TextGen and is available as a separate tab in concept's editor. MPS provides destructive update of generated assets only.

For instance, baseLanguage's TextGen aspect generates *.java files at the following location:
<generator output path>\<model name>\<ClassName>.java
where:
Generator output path - is specified in the module, which owns the input model (see MPS modules).
Model name - is a path segment created by replacing '.' with the file separator in the input model's name.

Icon

For a quick how-to document on the MPS generator please check out the Generator Cookbook.

Overview

Generator Module

Unlike any other language aspect, the generator aspect is not a single model. Generator specification can comprise many generator models as well as utility models. A Generator Model contains templates, mapping configurations and other constructions of the generator language.

A Generator Model is distinguished from a regular model by the model stereotype - 'generator' (shown after the model name as <name>@generator).
The screenshot below shows the generator module of the smodel language as an example.

Research bundled languages yourself

Icon

You can research the smodel (and any other) language generator by yourself:

  • download MPS (here);
  • create new project (can be empty project);
  • use the Go To -> Go to Language command in the main menu to navigate to the smodel language (its full name is jetbrains.mps.lang.smodel)

Creating a New Generator

A new generator is created by using the New -> Generator command in the language's popup menu.

Technically, it is possible to create more than one generator for one language, but at the time of writing MPS does not provide full support for this feature. Therefore, languages normally have only one (or none) generator. For that reason, the generator's name is not important. Everywhere in the MPS GUI a generator module can be identified by its language name.

When creating a new generator module, MPS will also create the generator model 'main@generator' containing an empty mapping configuration node.

Generator Properties

As a module, generator can depend on other modules, have used languages and used devkits (see Module meta-information).

The generator properties dialog also has two additional properties:

Generating Generator

MPS generator engine (or the Generator language runtime) uses mixed compilation/interpretation mode for transformation execution.

Templates are interpreted and filled at runtime, but all functions in rules, macros, and scripts must be pre-compiled.

(lightbulb) To avoid any confusion, always follow this rule: after any changes made to the generator model, the model must be re-generated (Shift+F9). Even better is to use Ctrl+F9, which will re-generate all modified models in the generator module.

Transformation

The transformation is described by means of templates. Templates are written using the output language and so can be edited with the same cell editor that would normally be used to write 'regular code' in that language. Therefore, without any additional effort the 'template editor' has the same level of tooling support right away - syntax/error highlighting, auto-completion, etc. The templates are then parametrized by referencing into the input model.

The applicability of individual templates is defined by #Generator Rules, which are grouped into #Mapping Configurations.

Mapping Configurations

A Mapping Configuration is a minimal unit, which can form a single generation step. It contains #Generator Rules, defines mapping labels and may include pre- and post-processing scripts.

Generator Rules

Applicability of each transformation is defined by generator rules.
There are six types of generator rules:

  • conditional root rule
  • root mapping rule
  • weaving rule
  • reduction rule
  • pattern rule
  • abandon root rule
  • drop attribute rule (new in 3.3)

Each generator rule consists of a premise and a consequence (except for the abandon root rule and drop attribute rule, with predefined consequence that cannot be specified by the user).

All rules except for the conditional root rule contain a reference to the concept of the input node (or just input concept) in its premises. All rule premises also contain an optional condition function.

Rule consequence commonly contains a reference to an external template (i.e. a template declared as a root node in the same or different model) or so-called in-line template (conditional root rule and root mapping rule can only have reference to an external template). There are also several other versions of consequences.

The following screenshot shows the contents of a generator model and a mapping configuration example.

Macros

The code in templates can be parameterized through macros. The generator language defines three kinds of macros:

  • property macro - computes a property value;
  • reference macro - computes the target (node) of a reference;
  • node macro - is used to control template filling at generation time. There are several versions of node macro - LOOP-macro is an example.

Macros implement a special kind of so-called annotation concept and can wrap property, reference or node cells (depending on the kind of macro) in the template code.

Code wrapping (i.e. the creation of a new macro) is done by pressing Ctrl+Shift+M or by applying the 'Create macro' intention.

The following screenshot shows an example of a property macro.

Macro functions and other parameterization options are edited in the inspector view. Property macro, for instance, requires specifying the value function, which will provide the value of the property at generation time. In the example above, output class node will get the same name that the input node has.

The node parameter in all functions of the generator language always represents the context node to which the transformation is currently being applied (the input node).

Some macros (such as LOOP and SWITCH-macro) can replace the input node with a new one, so that subsequent template code (i.e. code that is wrapped by those macros) will be applied to the new input node.

External Templates

External templates are created as a root node in the generator model.

There are two kinds of external templates in MPS.

One of them is root template. Any root node created in generator model is treated as a root template unless this node is a part of the generator language (i.e. mapping configuration is not a root template). Root template is created as a normal root node (via Create Root Node menu in the model's popup).

The following screenshot shows an example of a root template.

This root template will transform input node (a Document) into a class (baseLanguage). The root template header is added automatically upon creation, but the concept of the input node is specified by the user.

(lightbulb) It is a good practice to specify the input concept, because this allows MPS to perform static type checking in the code of the macro function.

A Root template (reference) can be used as a consequence in conditional root rules and root mapping rules. ((warning) When used in a conditional root rule, the input node is not available).

The second kind of template is defined in the generator language and its concept name is 'TemplateDeclaration'. It is created via the 'template declaration' action in the Create Root Node menu.

The following screenshot shows an example of template declaration.

The actual template code is 'wrapped' in a template fragment. Any code outside template fragment is not used in transformation and serves as a context (for example you can have a Java class, but export only one of its method as a template).

Template declaration can have parameters, declared in the header. Parameters are accessible through the #generation context.

Template declaration is used in consequence of weaving, reduction and pattern rules. It is also used as an included template in INCLUDE-macro (only for templates without parameters) or as a callee in CALL-macro.

Template Switches

A template switch is used when two or more alternative transformations are possible in a certain place in template code. In that case, the template code that allows alternatives is wrapped in a SWITCH-macro, which has reference to a Template Switch. Template Switch is created as a root node in the generator model via the Create Root Node menu (this command can be seen in the 'menu' screenshot above).

The following screenshot shows an example of a template switch.


Generator Language Reference

Mapping Configuration

Mapping Configuration is a container for generator rules, mapping label declarations and references to pre- and post-processing scripts. A generator model can contain any number of mapping configurations - all of them will be involved in the generation process, if the owning generator module is involved. Mapping configuration also serves as a minimal generator unit that can be referenced in the mapping priority rules (see Generation Process: Defining the Order of Priorities).

Generator Rule

Generator Rule specifies a transformation of an input node to an output node (except for the conditional root rule which doesn't have an input node and simply creates a new node in the output model). All rules consist of two parts - a premise and a consequence (except for the abandon root rule, which doesn't have a consequence and simply ignores the input node). Any generator rule can be tagged by a mapping label.

All generator rules' functions have the following parameters:

  • node - the current input node (all except the condition-function in conditional root rule)
  • genContext - generation context - allows searching for output nodes, generating of unique names and others (see #generation context)

Generator Rules:

Rule

Description

Premise

Consequence

conditional root rule

Generates a root node in the output model. Applied only one time (max) during a single generation step.

condition function (optional), missing condition function is equivalent to a function always returning true.

root template (ref)

root mapping rule

Generates a root node in the output model.

concept - applicable concept (concept of the input node)
inheritors - if true then the rule is applicable to the specified concept and all its sub-concepts. If false (default) then the sub-concepts are not applicable.
condition function (optional) - see conditional root rule above.
keep input root - if false then the input root node (if it's a root node) will be dropped. If true then input root will be copied to the output model.

root template (ref)

weaving rule

Allows to insert additional child nodes into the output model. Weaving rules are processed at the end of a generation micro-step just before map_src and reference resolving. The rule is applied on each input node of the specified concept. The parent node for insertion should be provided by the context function.
(see #Model Transformation)

concept - same as above
inheritors - same as above
condition function (optional) - same as above

  • external template (ref)
  • weave-each 
  • context - function that computes the (parent) output node into which the output node(s) generated by this rule will be inserted.
  • anchor (available in Inspector) - specifies a node within the context collection, in front of which the nodes should be inserted, null means insert at the end of the collection

reduction rule

Transforms the input node while this node is being copied to the output model.

concept - same as above
inheritors - same as above
condition function (optional) - same as above

  • external template (ref)
  • in-line template
  • in-line switch
  • dismiss top rule
  • abandon input

pattern rule

Transforms the input node, which matches the pattern.

pattern - pattern expression
condition function (optional) - same as above

  • external template (ref)
  • in-line template
  • dismiss top rule
  • abandon input

abandon root rule

Allows to drop an input root node which otherwise would be copied into the output model.

applicable concept ((warning) including all its sub-concepts)
condition function (optional) - same as above

n/a

drop attribute ruleFor a transformed node, controls which attributes get copied from the input nodeconcept - concept of the attribute node (subconcept of jetbrains.mps.lang.core.structure.Attribute)
inheritors - if true then the rule is applicable to the specified concept and all its sub-concepts. If false (default) then the sub-concepts are not applicable.
condition function (optional) - further restrictions to trigger the rule

n/a

Attribute is not copied when the rule matches

Rule Consequences:

Consequence

Usage

Description

root template (ref)

  • conditional root rule
  • (root) mapping rule

Applies the root template

external template (ref)

  • weaving rule
  • reduction rule
  • pattern rule

Applies an external template. Parameters should be passed if required, they can be one of:

  • pattern captured variable (starting with # sign)
  • integer or string literal
  • null, true, false
  • query function

weave-each

weaving rule

Applies an external template to a set of input nodes.
Weave-each consequence consists of:

  • foreach function - returns a sequence of input nodes
  • reference on an external template

in-line template

  • reduction rule
  • pattern rule

Applies the template code which is written right here.

in-line switch

reduction rule

Consists of set of conditional cases and a default case.
Each case specify a consequence, which can be one of:

  • external template (ref)
  • in-line template
  • dismiss top rule
  • abandon input

dismiss top rule

  • reduction rule
  • pattern rule

Drops all reduction-transformations up to the point where this sequence of transformations has been initiated by an attempt to copy the input node to the output model. The input node will be copied 'as is' (unless some other reduction rules are applicable). The user can also specify an error, warning or an information message.

abandon input

  • reduction rule
  • pattern rule

Prevents the input node from being copied into the output model.

Root Template

Root Template is used in conditional root rules and (root) mapping rules. The Generator language doesn't define specific concept for root template. The generator language defines a special kind of annotation - root template header, which is automatically added to each new root template. The root template header is used for specifying an expected input concept (i.e. concept of input node). MPS uses this setting to perform a static type checking of code in various macro-functions, which are used in the root template.

Icon

Technically, any root node in the output language can be considered to be a Root Template, when created in the generator model. However, root nodes without annotations are treated by the generator as auxiliary, utility nodes e.g. to specify reference targets and thus are not considered for target language evaluation. A warning is displayed and a quick fix is offered to add the root template header annotation to the root template that miss one. We highly recommend language designers to fix their templates and to always create new root templates with the annotation included.


 

External Template

External Template is a concept defined in the generator language. It is used in weaving rules and reduction rules.

In external templates the user specifies the template name, input concept, parameters and a content node.

The content node can be any node in the output language. The actual template code in external templates is surrounded by template fragment 'tags' (the template fragment is also a special kind of annotation concept). The code outside template fragment serves as a framework (or context) for the real template code (template fragment) and is ignored by the generator. In external template for weaving rule, the template's context node is required (it is a design-time representation of the rule's context node), while the template for reduction rule can be just one context-free template fragment. An external template for a reduction rule must contain exactly one template fragment, while a weaving rule's template can contain more than one template fragments.

Template fragment has a mapping label property, which is edited in Inspector view.

Mapping Label

Mapping Labels are declared in a mapping configuration and references stored to this declaration are used to label generator rules, macros and template fragments. Such marks allow finding of an output node by a known input node (see #generation context).

Properties:

  • name
  • input concept (optional) - expected concept of the input node of the transformation performed by the tagged rule, macro or template fragment
  • output concept (optional) - expected concept of the output node of the transformation performed by the tagged rule, macro or template fragment

MPS makes use of the input/output concept settings to perform static type checking in get output ... operations (see #generation context).

Macro

Macro is a special kind of an annotation concept which can be attached to any node in template code. Macro brings dynamic aspect into otherwise static template-based model transformation.

Property- and reference-macros are attached to property- and reference-cells respectively, while node-macros (which come in many variations - LOOP, IF etc.) are attached to cells representing the whole node in the cell-editor. All properties of a macro are edited using the inspector view.

All macros have the mapping label property - reference to a mapping label declaration. Additionally, all macros can be parameterized by various macro-functions - depending on the type of the macro. Any macro-function has at least the three following parameters:

  • node - the current input node;
  • genContext - generation context - allows searching for output nodes, generating of unique names and others;
  • operationContext - instance of jetbrains.mps.smodel.IOperationContext interface (used rarely).

Many types of macros have the mapped node (or mapped nodes) function. This function computes the new input node - a substitution for the current input node. If the mapped node function returns null or the mapped nodes function returns an empty sequence, then the generator will skip this macro altogether. I.e. no output will be generated in this place.

Macro

Description

Properties (if not mentioned above)

Property macro

Computes the value of a property.

value function:

  • return type - string, boolean or int - depending on the property type.
  • parameters - standard + templateValue - value in the template code wrapped by the macro.

Reference macro

Computes the referent node in the output model.
Normally executed at the end of a generation micro-step, when the output model (tree) is already constructed.
Can also be executed earlier, if the user code is trying to obtain the target of the reference.

Reference macro supports SNodeReference as specification of a new target. With that, templates don't need access to a target's node model the moment they are applied. 

referent function:

  • return type
    • node (type depends on the reference link declaration)
    • SNodeReference
    • string identifying the target node (see note).
  • parameters - standard + outputNode - source of the reference link (in the output model).

IF

The wrapped template code is applied only if the condition is true. Otherwise the template code is ignored and an 'alternative consequence' (if any) is applied.

condition function
alternative consequence (optional) - any of:

  • external template (ref)
  • in-line template
  • abandon input
  • dismiss top rule

LOOP

Computes new input nodes and applies the wrapped template to each of them.

mapped nodes function

INCLUDE

The wrapped template code is ignored (it only serves as an anchor for the INCLUDE-macro), a reusable external template will be used instead.

Null input makes INCLUDE effectively a no-op.

mapped node function (optional)
include template - reference to a reusable external template

CALL

Invokes template and replaces wrapped template code with the result of template invocation. Supports templates with parameters.

Null input node is tolerated, and the template is ignored altogether in this case, i.e. CALL yields empty collection of nodes as a result when input/mapped node is null.

mapped node function (optional)
call template - reference to a reusable external template

argument - one of

  • pattern captured variable
  • integer or string literal
  • null, true, false
  • query function

SWITCH

Provides a way to many alternative transformations in the given place in the template code.
The wrapped template code is applied, if none of switch cases is applicable and no default consequence is specified in #template switch.

For null input node, SWITCH may react with a message (specified along with its rules), anchor template node is ignored, and SWITCH macro yields no results.

mapped node function (optional)
template switch - reference to a template switch

COPY-SRC

Copies an input node to the output model. The wrapped template code is ignored.

mapped node function - computes the input node to be copied.

COPY-SRCL

Copies input nodes to the output model. The wrapped template code is ignored.
Can be used only for children with multiple aggregation cardinality.

mapped nodes function - computes the input nodes to be copied.

MAP-SRC

Multifunctional macro, can be used for:

  • marking a template code with a mapping label;
  • replacing the current input node with a new one;
  • perform a non-template based transformation;
  • accessing the output node for some reason.
    The MAP-SRC macro is executed at the end of a generator micro-step - after all node- and property-macros, but before any reference-macro is run.

mapped node function (optional)
mapping func function (optional) - performes non-template based transformation.
If defined then the wrapped template code will be ignored.
Parameters: standard + parentOutputNode - parent node in the output model.
post-processing function (optional) - give access to the output node.
Parameters: standard + outputNode

MAP-SRCL

Same as MAP-SRC but can handle many new input nodes (similar to the LOOP-macro)

mapped nodes function
mapping func function (optional)
post-processing function (optional)

WEAVE

Allows to insert additional child nodes into the output model in a similar way Weaving rules are used. The node wrapped in the WEAVE macro (or provided by the use input function) will have the supplied template applied to it and the generated nodes will be inserted.

use input a function returning a collection of nodes to apply the macro to
weave reference to a template to weave into the nodes supplied as the input

Note

Icon

Reference resolving by identifier is only supported in BaseLanguage.
The identifier string for classes and class constructors may require (if class is not in the same output model) package name in square brackets preceding the class name:
[package.name]ClassName

Template Switch

A template switch is used in pair with the SWITCH-macro (the TemplateSwitchMacro concept). A single template switch can be re-used in many different SWITCH-macros. A template switch consists of set of cases and one default case. Each switch case is a reduction rule, i.e. a template switch contains a list of reduction rules (see #reduction rule).

The default case consequence can be one of:

  • external template (ref)
  • in-line template
  • abandon input
  • dismiss top rule
    .. or can be omitted. In this case the template code surrounded by corresponding SWITCH-macro will be applied.

A template switch can inherit reduction rules from other switches via the extends property. When the generator is executing a SWITCH-macro it tries to find most specific template switch (available in scope). Therefore the actually executed template switch is not necessarily the one that is defined in the template switch property of the SWITCH-macro.

Through the null-input message property the user can specify an error, warning or info message, which will be shown in the MPS messages view in case when the mapped node function in SWITCH-macro returns null (by default no messages are shown and the macro is skipped altogether).

A template switch can accept parameters, the same way as template declarations. A use of parametrized switch mandates arguments to be supplied in the SWITCH macro. The TemplateSwitchMacro concept supports switches both with and without arguments.

Generation Context (operations)

Generation context (the genContext parameter in macro- and rule-functions) allows finding of nodes in the output model, generating unique names and provides other useful functionality.

Generation context can be used not only in the generator models, but also in utility models - as a variable of type gencontext.

Operations of genContext are invoked using the familiar dot-notation: genContext.operation

Finding Output Node

get output <mapping label> for model ( <model> )

Returns the output node generated by a labeled conditional root rule in a specified model.
Issues an error, if there is more than one matching output node.

get output <mapping label> for ( <input node> )

Returns the output node generated from the input node by a labeled generator rule, a macro or a template fragment.
Issues an error if there is more than one matching output node.

pick output <mapping label> for ( <input node> )

(warning) only used in the context of the referent function in a reference-macro and only if the required output node is a target of the reference, which is being resolved by that reference-macro.
Returns the output node generated from the input node by a labeled generator rule, a macro or a template fragment. The difference between this and the previous operation is that this operation can automatically resolve the many-output-nodes conflict - it picks the output node, which is visible in the given context (see search scope).

get output list <mapping label> for ( <input node> )

Returns a list of output nodes generated from the input node by a labeled generator rule, a macro or a template fragment.

get copied output for ( <input node> )

Returns the output node, which has been created by copying of an input node. If during the copying, the input node has been reduced, but the concept of the output node is the same (i.e. it wasn't reduced into something totally different), then this is still considered 'copying'.
Issues an error if there is more than one matching output node.

Generating Unique Name

unique name from <base name> in context <node>

The uniqueness is secured throughout the whole generation session.
(warning) Clashing with names that weren't generated using this service is still possible.

The context node is optional, though we recommend to specify it to guarantee generation stability. If specified, then MPS tries its best to generated names 'contained' in a scope (usually a root node). Then when names are re-calculated (due to changes in the input model or in the generator model), this won't affect other names outside the scope.

Template Parameters

#patternvar

Value of the captured pattern variable
(warning) available only in rule consequences

param

Value of the template parameter
(warning) available only in external templates

Getting Contextual Info

inputModel

The current input model

originalModel

The original input model

outputModel

The current output model

templateNode

The template code surrounded by the macro.
It is only used in macro-functions

(warning) This operation has been deprecated in MPS 3.3.

The primary flaw is that this operation implies interpreted templates. There is no template model when templates are generated.

Besides, contract of the operation is vague (i.e. what does it give in a context of argument query for a template call).

get prev input <mapping label>

Returns the input node that has been used for enclosing the template code surrounded by the labeled macro.
It is only used in macro-functions.

Transferring User Data

During generation MPS maintains three maps of user objects, each of which has different life span:

  • session object - kept throughout the whole generation session;
  • step object - kept through a single generation step;
  • transient object - only alive during a micro step.

The developer can access the user object maps using the array (square brackets) notation:

The key can be any object (java.lang.Object).

binding user data with particular node

Icon

The session- and step-object cannot be used to pass data associated with a particular input node across steps and micro-steps, because neither an input node nor its id can serve as a key (output nodes always have a different id).
To pass such data you should use the putUserObject and getUserObject methods defined in class jetbrains.mps.smodel.SNode.
The data will be transferred to all output copies of the input node. The data will be also transferred to the output node, if a slight reduction (i.e. without changing of the node concept) took place while the node was being coppied.

Logging

Creates message in the MPS message view. If the node parameter is specified then clicking on the message will navigate to that node. In case of an error message, MPS will also output some additional diagnostic info.

Utilities (Re-usable Code)

If you have duplicated code (in rules, macros, etc.) and want to say, extract it to re-usable static methods, then you must create this class in a separate, non-generator model.

If you create an utility class in the generator model (i.e. in a model with the 'generator' stereotype), then it will be treated as a root template (unused) and no code will be generated from it.

Mapping Script

A Mapping script is user code, which is executed either before a model transformation (pre-processing script) or after it (post-processing script). It should be referenced from #Mapping Configuration to be invoked as a part of it's generation step. Mapping scripts provide the ability to perform non-template based model transformations.

Pre-processing scripts are also commonly used for collecting certain information from input model that can be later used in the course of template-based transformation. The information collected by script is saved as a transient-, step- or session-object (see generation context).

Script sample:

Properties:

script kind

  • pre-process input model - the script is executed at the beginning of a generation step, before the template-based transformation;
  • post-process output model - the script is executed at the end of a generation step, after the template-based transformation.

modifies model

only available if script kind = pre-process input model
If set true and the input model is the original input model, then MPS will create a transient input model before applying the script.
If set false but the script tries to modify the input model, then MPS will issue an error.

Code context:

model

The current model

genContext

The generation context to give access to transient/session or step objects.

invocation context

Operation context (jetbrains.mps.smodel.IOperationContext java interface) associated with the module - the owner of the original input model

(warning) Operation context has been deprecated will be removed in the next release, please don't use.


The Generator Algorithm

The process of generation of target assets from an input model (generation session) includes 5 stages:

  • Defining all generators that must be involved
  • Defining the order of priorities of transformations
  • Step-by-step model transformation
  • Generating text and saving it to a file (for each root in output model)
  • Post-processing assets: compiling, etc.

We will discuss the first three stages of this process in detail.

Defining the Generators Involved

To define the required generators, MPS examines the input model and determines which languages are used in it. Doing this job MPS doesn't make use of 'Used Languages' specified in the model properties dialog. Instead MPS examines each node in the model and gathers languages that are actually used.

From each 'used language' MPS obtains its generator module. If there are more than one generator module in a language, MPS chooses the first one (multiple generators for the same language are not fully supported in the current version of MPS). If any generator in this list depends on other generators (as specified in the 'depends on generators' property), then those generators are added to the list as well.

After MPS obtains the initial list of generators, it begins to scan the generator's templates in order to determine what languages will be used in intermediate (transient) models. The languages detected this way are handled in the same manner as the languages used in the original input model. This procedure is repeated until no more 'used languages' can be detected.

Explicit Engagement

In some rare cases, MPS is unable to detect the language whose generator must be involved in the model transformation. This may happen if that language is not used in the input model or in the template code of other (detected) languages. In this case, you can explicitly specify the generator engagement via the Languages Engaged on Generation section in the input model's properties dialog (Advanced tab).

Dependency scope/kind - 'Generation Target' and 'Design'.

'Generation Target' replaces 'Extends' relation between two languages (L2 extends L1), when one needed to specify that Generator of L2 generates into L1 and thus needs its runtime dependencies. Now, when a language (L2) is translated to another language (L1), and L1 has runtime dependencies, use L1 as 'Generation Target' of L2. Though this approach is much better than 'Extends', it's still not perfect as it's rather an attribute of a generator than of a language. Once Generators become fully independent from their languages, we might need to fix this approach (different generators may target different languages, thus target has to be specified for a generator, not the source language).

'Design' dependency replaces 'Extends' between two generators. Use it when you need to reference another generator to specify priority rules (though consider if you indeed need these priorities, see changes in the Generator Plan, below)

Defining the Order of Priorities

As we discussed earlier, a generator module contains generator models, and generator models contain mapping configurations. Mapping configuration (mapping for short) is a set of generator rules. It is often required that some mappings must be applied before (or not later than, or together with) some other mappings. The language developer specifies such a relationship between mappings by means of mapping constraints in the generator properties dialog (see also #Mapping Priorities and the Dividing Generation Process into Steps demo).

After MPS builds the list of involved generators, it divides all mappings into groups, according to the mapping priorities specified. All mappings for which no priority has been specified fall into the last (least-priority) group.

tip

Icon

You can check the mapping partitioning for any (input) model by selecting Show Generation Plan action in the model's popup menu.
The result of partitioning will be shown in the MPS Output View.

Optimized Generation Plan

When planning the generation phase, MPS prefers to keep every generator as lonely as possible. Eventually, you'll see many relatively small and fast to process generation steps. Of course, the generators forced to run together with priority rules still run at the same step. Handling several unrelated generators at the same generation step (MPS prior to 3.2) proved to be inefficient, since it imposed a lot of unnecessary checking for rule applicability across other generators from the same step. With in-place transformation in 3.2 and later, the performance penalty for each extra generation steps is negligible.

Ignored priority rules

In addition to conflicting priorities, there are rules that get ignored during the generation plan. This might happen if an input model doesn't have any concept of a language participating in a priority rule. Since there's no actual use of a language, the rule is ignored, and the 'Show Generation Plan' action reports them along with conflicting rules. Previous MPS versions used to include generators of otherwise unused languages into the generation process, now these generators get no chance to jump in.

Implicit priorities

Target languages (languages produced by templates) are considered as implicit 'not later than' rules. You don't need to specify these priorities manually. MPS automatically inserts "not later than" rules for all generator models in the source and target languages. It is important to understand that priority rules work on the model granularity level.

Icon

This implicit priority rule between two generator models is ignored if an explicit priority rule is defined for these two models, one from the language that generates into the other language and one from the other language.

Model Transformation

Each group of mappings is applied in a separate generation step. The entire generation session consists of as many generation steps as there were mapping groups formed during the mapping partitioning. A generation step includes three phases:

  • Executing pre-mapping scripts
  • Template-based model transformation
  • Executing post-mapping scripts

The template-based model transformation phase consists of one or more micro-steps. A micro-step is a single-pass model transformation of an input model into a transient (output) model.

While executing micro-step MPS follows the next procedure:

  1. Apply conditional root rules (only once - on the 1-st micro-step)
  2. Apply root mapping rules
  3. Copy input roots for which no explicit root mapping is specified (this can be overridden by means of the 'keep input root' option in root mapping rules and by the 'abandon root' rules)
  4. Apply weaving rules
  5. Apply delayed mappings (from MAP_SRC macro)
  6. Revalidate references in the output model (all reference-macro are executed here)

There is no separate stage for the application of reduction and pattern rules. Instead, every time MPS copies an input node into the output model, it attempts to find an applicable reduction (or pattern) rule. MPS performs the node copying when it is either copying a root node or executing a COPY_SRC-macro. Therefore, the reduction can occur at either stage of the model transformation.

MPS uses the same rule set (mapping group) for all micro-steps within the generation step. After a micro-step is completed and some transformations have taken place during its execution, MPS starts the next micro-step and passes the output model of the previous micro-step as input to the next micro-step. The whole generation step is considered completed if no transformations have occurred during the execution of the last micro-step, that is, when there are no more rules in the current rule set that are applicable to nodes in the current input model.

The next generation step (if any) will receive the output model of previous generation step as its input.

tip

Icon

Intermediate models (transient models) that are the output/input of generation steps and micro-steps are normally destroyed immediately after their transformation to the next model is completed.
To keep transient models, enable the following option:
Settings -> Generator Settings -> Save transient models on generation

See also:

Handling of node attributes during generation

Node attributes constitute generic extension mechanism, thus Generator shall preserve attributes along the transformation process (unless attribute designer opts not to keep them) without explicit support in any template. When an input node is transformed to another node, Generator copies attributes of the input node to the output. Copy is controlled by newly introduced drop attribute rule only, and happens regardless of @attribute info specification (i.e. its attributed concept or multiple restrictions). The fact that transformation rule may have produced attribute node itself is not taken into account (i.e. if a reduction rule explicitly copies node attributes to a newly created output node, the attributes would get duplicated due to automatic copy of attributes. However, it's rare for a reduction rule to copy node attributes, and the issue, if ever shows up, is easy to mitigate with drop rules).

While copying attributes of a node, Generator consults drop attribute rules (newly introduced in MPS 3.3, reside next to abandon root rules) to see if language designer don't need these attributes to survive transformation process. This rules are quite similar to abandon root rules - when any rule is triggered, attribute is not copied into output model.

With the growing adoption of attributes and their increasing complexity, we enabled the generator to transform the attribute contents using the regular template processing rules:

  • references to nodes in the same model get updated to point to the respective nodes in the output model
  • reduction rules are applied in order to transform children of the attribute node.

In-place transformation

Generators for languages employed in a model are applied sequentially (aka Generation Plan). Effectively, each generation step modifies just a fraction of original model, and the rest of the model is copied as-is. With huge models and numerous generation steps this approach proves to be quite ineffective. In-place transformation tries to address this with a well-known 'delta' approach, where changes only are collected and applied to original model to alter it in-place.

In version 3.1 in-place transformation is left as an option, enabled by default and configurable through the Project settings -> Generator. Clients are encouraged to fix their templates that fail in the in-place mode, as in-place generation is likely to become the only generation mode later down the road.

Use of in-place transformation brings certain limitations or might even break patterns that used to work in the previous MPS versions:

  • Most notable and important - there's no output model at the moment when rule's queries/conditions are executed. To consult the output model during the transformation process is a bad practice, and in-place transformation enforces removing it. Access to the output model from a transformation rule implies certain order of execution, thus effectively limiting the set of optimizations applicable by the MPS generator. The contract of a transformation rule, with a complete input and a fraction of the output that this particular rule is responsible for, is more rigorous than "a complete input model" and "an output model in some uncertain state".
  • The output model is indeed there for weaving rules, as their purpose is to deal with output nodes.
  • The process of delta building requires the generator to know about the nodes being added to the model. Thus, any implicit changes in the output model that used to work would fail with in-place generation enabled. As an example, consider MAP-SRC with a post-process function, which replaces the node with a new one:postprocess: if (node.someCondition()) node.replace with new(AnotherNode);. Generator records a new node produced by MAP-SRC, schedules it for addition, and delays post-processing. Once post-processing is over, there's no way for the generator to figure out the node it tracks as 'addition to output model' is no longer valid and there's another node which should be used instead. Of course, the post-process can safely alter anything below the node produced by MAP-SRC, but an attempt to go out from the sandbox of the node would lead to an error.
  • Presence of active weaving rules prevents in-place transformation as these rule require both input and output models.

Generation trace

Much like in-place transformation, the updated generation trace is inspired by the idea to track actual changes only. Now it's much less demanding, as only the transformed nodes are being tracked. Besides, various presentation options are available to give different perspective on the transformation process.


Mapping Priorities

Mapping priorities are a set of mapping priority rules specifying an order of priority between sets of generator rules (mapping configurations).

Mapping priorities are specified in generator module property dialog.
See also: Generator Properties, Generation Process: Defining the Order of Priorities, Demo 6: Dividing Generation Process into Steps

Each mapping priority rule consists of left part, right part and priority symbol in the middle.
For instance:

Left part of priority rule can only refer to mapping configurations in this generator.

Notation

Refers to

*

All mappings in this generator

modelName . *

All mappings in model

modelName . mappingName

Mapping configuration

{ ... }

Set of mappings.
An entry can be either
modelName . * or
modelName . mappingName

Right part of priority rule can refer to mapping configurations in this generator as well as other generators visible in scope (see Generator Properties: dependency on other generators).

Notation

Refers to

*

All mappings in this generator

modelName . *

All mappings in model

modelName . mappingName

Mapping configuration

{ ... }

Set of mappings.
An entry can be either
modelName . * or
modelName . mappingName

[languageName / generatorName : * ]

All mapping in external generator

[languageName / generatorName : modelName]

All mapping in external generator model

[languageName / generatorName : modelName . mappingName]

External mapping configuration

[languageName / generatorName : { ... }]

All mapping in external generator

[*:*]

All possible mappings

Priority symbol

Notation

Description

<

Mappings in left part are processed before mappings in right part

<=

Mappings in left part are processed not later than mappings in right part

==

Mappings in left part and right part are processed together


Generation plan

Generation plans allow developers to specify the desired order of generation for their models explicitly and thus gain better control over the generation process.

Motivation

Specifying mutual generator priorities may become cumbersome for larger projects. Additionally, in order to specify the priorities the involved languages need to know about one another by declaring appropriate mutual dependencies, which breaks their (sometimes desired) independence. Generation plans put the responsibility of proper ordering of generation steps into a single place - the generation plan. They allow language designers to provide intuitive means for end-user models to be processed in a desired order. Generation plans list all the languages that should be included in the generation process, order them appropriately and optionally specify checkpoints, at which the generator shall preserve the current transient models. These models can then be used for automatic cross-model reference resolution further down the generation process.

 

Icon

The mechanism described here is a preliminary implementation that is likely to be further evolved in the following MPS versions. The general direction is:

  • to give more flexibility to the language designers in how they organize builds for users of their languages
  • to relieve the end-developers (aka language users) from the need to handle directly build plans, genplan models or build facets

So the actual end-user mechanism is likely to become more dependent on the language designer's intentions. The approach described here, based on facet and a genplan model, is a mere sample of how to accomplish custom generation.

Defining a generation plan

In order to create a generation plan, you first need to create a model. You may consider giving the model a genplan stereotype to easily distinguish it from ordinary models, but this is not mandatory.

After importing the jetbrains.mps.lang.generator.plan and jetbrains.mps.lang.smodel languages, you can create root node of the Plan concept, which will represent your generation plan:

The generation plan consists of transforms and checkpoints.

It is also possible to specify the required generators explicitly.

Transforms represent generation steps and include languages that should be generated as part of that generation phase.

Apply represents an explicit invocation of a particular generator.  The apply with extended statement applies in a single step the specified generators and those that extend them. This allows the language designer to accommodate for possible extensions.

 

Checkpoints represent points during the generation, at which the intermediate models should be preserved. References that will be resolved later in the generation will be able to look-up nodes in the stored intermediate models for their resolution through mapping labels. You can view these checkpoint models in the Project View tool window:

<TODO - may be this is no longer valid> These intermediate checkpoint models are preserved until you shutdown MPS or until you rebuild the models or the models that they depend on. Alternatively you can remove them manually:

Checkpoints provide synchronization points between different plans. The checkpoint models are denoted with a stereotype that matches the name of a checkpoint the model has been created with. Models are persisted alongside the generated sources using the naming scheme of <plan-name>-<checkpoint name>. 

Distinct statements allow for capturing different aspects of a checkpoint.

  • declare checkpoint <name> statement - specify a label that generator plans could share among themselves. This statement does not record/persist the state of the transformed model, it is a mere declaration that other generation plans will be able to refer to.
  • checkpoint <checkpoint> - records/persists the state of the transformed model. It can either declare a checkpoint in-place or refer to a declared checkpoint.
  • synchronize with <checkpoint> statements -  instructs the generation plan to look up the target nodes in persisted models of the specified checkpoint, but do not persist its own nodes (read-only access to the check-point). This statement doesn't introduce any new state, but references a checkpoint declared elsewhere.

Specifying a generation plan for models

Modules that should have their models built following a generation plan need to enable the Custom generation facet and point to the actual desired generation plan:

 

Verifying the generation plan

The Show generation plan action in the models' pop-up menu will correctly take the generation plan into account when building the outline for the generation:

To view the original, on generator priorities based, generation plan that would be used without the explicit generation plan script, hold the Alt key while clicking the Show Generation Plan menu entry:

Note that the report states in the header that it is not the currently active plan.

Using DevKits to associate a generation plan

DevKits can associate a generation plan, as well.

First add dependencies on languages and solutions that the DevKit should be wrapping. Then specify the Generation plan from within the imported solutions, which will be associated with the DevKit. Any model that imports that DevKit will get the DevKit's associated generation plan applied to it.

Icon

Only one DevKit with a plan is allowed per model at the moment.

Cross-model generation

Model is the unit of generation in MPS. All entities in a single model are generated together and references between the nodes can be resolved with reference macros and mapping labels. Mapping labels, however, are not by default accessible from other models. This complicates generation of references that refer to nodes from other models. Fortunately, regular mapping labels can support mutually independent generation of models with cross-references, if the models share a generation plan. The mechanism leverages checkpoints to capture the intermediate transient models and then use them for reference resolution.

In essence, to preserve cross-model references when generating multiple models, make sure that your models share a generation plan. That generation plan must define checkpoints at moments, when the mapping labels that are used for cross-model reference resolution have been populated. The rest will be taken care of automatically. The reference macros can resolve nodes from mapping labels through the usual genContext.get output by label and input (for nodes generated through reduction or root mapping rules) or genContext.get output for model (for nodes generated through conditional root mapping rules) ways.

 

Linking checkpoint models

 

Models created at checkpoints now keep a reference to the previous checkpoint model in the sequence. This helps the Generator discover mapped nodes matching input that spans several generator phases.

Debug information in the checkpoint models

To ease debug of cross-model generation scenarios, a dedicated root inside each checkpoint model lists the mapping label names along with pointers to the stored input and output nodes. Investigation of the mapping labels exposed at each checkpoint can substantially help debugging cross-model generation scenarios and fix unresolved references. Thus, next time your cross-model reference doesn't resolve, inspect corresponding checkpoint model to see if there's indeed a label for your input.

Icon

A video on setting a generation plan for a solution as well as for a DevKit is available.

Generating language descriptor models

Generation plans have been enhanced to generate descriptor models for languages (known as <language.name>@descriptor). The structure, textgen, typesystem, dataflow and constraints aspects are now generated with generation plans and they use the new cross-model reference resolution mechanism.
Custom aspects defined by language authors can join the generation plan, as well. If you got a custom aspect, you should make sure that its generator extends the generator of jetbrains.mps.lang.descriptor language, as this is the way to get custom extensions activated for the plan.

 


Generating from Ant

The Ant MPS generator task exposes properties configurable from the build script (parallel, threads, in-place, warnings). The build language uses the Ant Generate task under the hood to transform models during the build process. This task now exposes parameters familiar from the Generator settings page:

  • strict generation mode
  • parallel generation with configurable number of threads
  • option to enable in-place transformation
  • option to control generation warnings/errors.

These options are also exposed at build language with the BuildMps_GeneratorOptions concept, so that build scripts have more control over the process.

Examples

If you're feeling like it's time for more practical experience, check out the generator demos.
The demos contain examples of usage of all the concepts discussed above.


Previous Next

  • No labels