Child pages
  • Developing Custom Language Plugins for IntelliJ IDEA

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • Syntax highlighting: The lexer is returned from the implementation of the SyntaxHighlighterFactory interface which is registed in the com.intellij.lang.syntaxHighlighterFactory extension point.
  • Building the syntax tree of a file: the lexer is expected to be returned from ParserDefinition.createLexer(), and the ParserDefinition interface is registered in the com.intellij.lang.parserDefinition extension point.
  • Building the index of the words contained in the file: If the lexer-based words scanner implementation is used, the lexer is passed to the DefaultWordsScanner constructor.

The lexer used for syntax highlighting can be invoked incrementally to process only the changed part of a file, whereas lexers used in other contexts are always called to process an entire file, or a complete language construction embedded in a file in a different language.

A lexer that can be used incrementally may need to return its state, which means the context corresponding to each position in a file. For example, a Java lexer could have separate states for top level context, comment context and string literal context. An important requirement for a syntax highlighting lexer is that its state must be represented by a single integer number (returned from Lexer.getState()). That state will be passed to the Lexer.start() method, along with the start offset of the fragment to process, when lexing is resumed from the middle of a file. Lexers used in other contexts can always return 0 from the getState() method.

The easiest way to create a lexer for a custom language plugin is to use JFlex. IDEA contains adapter classes (FlexLexer and FlexAdapter) that adapt JFlex lexers to the IDEA lexer API. The source code of IntelliJ IDEA Community Edition includes a patched version of JFlex 1.4.1 (tools/lexer/jflex-1.4) and lexer skeleton file (tools/lexer/idea-flex.skeleton) which can be used for creating lexers compatible with FlexAdapter. The patched version of JFlex provides a new command line option --charat which changes the JFlex generated code so that it works with the IDEA skeleton (which passes the source data for lexing as a CharSequence and not as an array of characters).

...

Types of tokens for lexers used in IDEA are defined by instances of IElementType. A number of token types common for all languages are defined in the TokenType interface; custom language plugins should reuse these token types wherever applicable. For all other token types, the plugin needs to create new IElementType instances and associate with the language in which the token type is used. The same IElementType instance should be returned every time a particular token type is encountered by the lexer.

...

An important feature which can be implemented at lexer level is mixing languages within a file (for example, embedding fragments of Java code in some template language). If a language supports embedding its fragments in another language, it needs to define the chameleon token types for different types of fragments which can be embedded, and these token types need to implement the ILazyParseableElementType interface. The lexer of the enclosing language needs to return the entire fragment of the embedded language as a single chameleon token, of the type defined by the embedded language. To parse the contents of the chameleon token, IDEA will call the parser of the embedded language through a call to ILazyParseableElementType.parseContents().

Implementing a Parser and PSI

Parsing files in IDEA is a two-step process. First, an abstract syntax tree (AST) is built, defining the structure of the program. AST nodes are created internally by IDEA and are represented by instances of the ASTNode class. Each AST node has an associated element type (IElementType instance), and the element types are defined by the language plugin. The top-level node of the AST tree for a file needs to have a special element type, implementing the IFileElementType interface.

The AST nodes have a direct mapping to text ranges in the underlying document (the bottom-most nodes of the AST match individual tokens returned by the lexer, and higher level nodes match multiple-token fragments). Operations performed on nodes of the AST tree (inserting, removing, reordering nodes and so on) are immediately reflected as changes to the text of the underlying document.

Second, a PSI (Program Structure Interface) tree is built on top of the AST, adding semantics and methods for manipulating specific language constructs. Nodes of the PSI tree are represented by classes implementing the PsiElement interface and are created by the language plugin in the ParserDefinition.createElement() method. The top-level node of the PSI tree for a file needs to implement the PsiFile interface, and is created in the ParserDefinition.createFile() method.

Example: ParserDefinition for Properties language

...

The base classes for the PSI implementation (PsiFileBase, the base implementation of PsiFile, and ASTWrapperPsiElement, the base implementation of PsiElement) are provided by IntelliJ IDEA. However, these classes are coupled to the internal implementation of IDEA and are located in idea.jar. Because of this, if you're using IntelliJ IDEA 10.5 or an earlier version to develop the plugin, you need to make sure that idea.jar is added to the list of JARs in the classpath of the IDEA SDK. (IntelliJ IDEA 11 adds idea.jar to the classpath automatically.)

...

The language plugin provides the parser implementation as an implementation of the PsiParser interface, returned from ParserDefinition.createParser(). The parser receives an instance of the PsiBuilder class, which is used to get the stream of tokens from the lexer and to hold the intermediate state of the AST being built. The parser must process all tokens returned by the lexer up to the end of stream (until PsiBuilder.getTokenType() returns null), even if the tokens are not valid according to the language syntax.

...

The parser works by setting pairs of markers (PsiBuilder.Marker instances) within the stream of tokens received from the lexer. Each pair of markers defines the range of lexer tokens for a single node in the AST tree. If a pair of markers is nested in another pair (starts after its start and ends before its end), it becomes the child node of the outer pair.

The element type for the marker pair (and for the AST node created from it) is specified when the end marker is set (by a call to PsiBuilder.Marker.done()). Also, it is possible to drop a start marker before its end marker has been set. The drop() method drops only a single start marker without affecting any markers added after it, and the rollbackTo() method drops the start marker and all markers added after it and reverts the lexer position to the start marker. These methods can be used to implement lookahead when parsing.

The method PsiBuilder.markerMarker.precede() is useful for right-to-left parsing when you don't know how many markers you need at a certain position until you read more input. For example, a binary expression a+b+c needs to be parsed as ( (a+b) + c ). Thus, two start markers are needed at the position of the token 'a', but that is not known until the token 'c' is read. When the parser reaches the '+' token following 'b', it can call precede() to duplicate the start marker at 'a' position, and then put its matching end marker after 'c'.

An important feature of PsiBuilder is its handling of whitespace and comments. The types of tokens which are treated as whitespace or comments are defined by the methods getWhitespaceTokens() and getCommentTokens() in the ParserDefinition class. PsiBuilder automatically omits whitespace and comment tokens from the stream of tokens it passes to PsiParser, and adjusts the token ranges of AST nodes so that leading and trailing whitespace tokens are not included in the node.

...

A number of functions which can be used for implementing and using the PSI can be found in the com.intellij.psi.util package, and in particular in the PsiUtil and PsiTreeUtil classes.

A very helpful tool for debugging the PSI implementation is the PsiViewer plugin. It can show you the structure of the PSI built by your plugin, the properties of every PSI element and highlight the text range of every PSI element.

Please see Indexing and PSI Stubs in IntelliJ IDEA for advanced topics.

Syntax Highlighting and Error Highlighting

The class used in IDEA to specify how a particular range of text should be highlighted is called TextAttributesKey. An instance of this class is created for every distinct type of item which should be highlighted (keyword, number, string and so on). The TextAttributesKey defines the default attributes which are applied to items of the corresponding type (for example, keywords are bold, numbers are blue, strings are bold and green). The mapping of the TextAttributesKey to specific attributes used in an editor is defined by the EditorColorsScheme class, and can be configured by the user if the plugin provides an appropriate configuration interface. Highlighting from multiple TextAttributeKey items can be overlaid - for example, one key may define an item's boldness and another its color.

The syntax and error highlighting is performed on multiple levels. The first level of syntax highlighting is based on the lexer output, and is provided through the SyntaxHighlighter interface. The syntax highlighter returns the TextAttributeKey instances for each token type which needs special highlighting. For highlighting lexer errors, the standard TextAttributeKey for bad characters (HighligherColors.BAD_CHARACTER) can be used.

Example: SyntaxHighlighlighter implementation for Properties language

The second level of error highlighting happens during parsing. If a particular sequence of tokens is invalid according to the grammar of the language, the PsiBuilder.error() method can be used to highlight the invalid tokens and display an error message showing why they are not valid.

The third level of highlighting is performed through the Annotator interface. A plugin can register one or more annotators in the com.intellij.annotator extension point, and these annotators are called during the background highlighting pass to process the elements in the PSI tree of the custom language. Annotators can analyze not only the syntax, but also the semantics of the text in the language, and thus can provide much more complex syntax and error highlighting logic. The annotator can also provide quick fixes to problems it detects.

...

To highlight a region of text as a warning or error, the annotator calls createErrorAnnotation() or createWarningAnnotation() on the AnnotationHolder object passed to it, and optionally calls registerFix() on the returned Annotation object to add a quick fix for the error or warning. To apply additional syntax highlighting, the annotator can call AnnotationHolder.createInfoAnnotation() with an empty message and then call Annotation.setTextAttributes() to specify the text attributes key for the highlighting.

...

Finally, if the custom language employs external tools for validating files in the language (for example, uses the Xerces library for XML schema validation), it can provide an implementation of the ExternalAnnotator interface and register it in com.intellij.externalAnnotator extension point. The ExternalAnnotator highlighting has the lowest priority and is invoked only after all other background processing has completed. It uses the same AnnotationHolder interface for converting the output of the external tool into editor highlighting.

...