Child pages
  • TokenRule

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migrated to Confluence 4.0
Code Block
syntax TokenRule = RuleAttributes "token" Name "=" RuleExpression SimpleRuleBody;

TokenRule is the same as SimpleRule with the exception of the following:

  • The rules declared through TokenRule describe a language token.
  • TokenRule type rules are automatically marked with the ExplicitSpaces attribute which indicates that whitespace rules should be placed manually.

TokenRule is specifically treated during error recovery. Besides, the parser will point at the rule, not at its content.

A token may also be described by RegexRule, however the structure of such rules won’t be accessible, which can compromise code highlighting and error recovery. As an example, comments and strings should be described with TokenRule, otherwise the parser will not be able to spot an unclosed string or comment.

For example:

Code Block
token Identifier = !Keyword IdentifierBody
{
  regex KeywordToken = "true" | "false" | "null";
  token Keyword = KeywordToken !IdentifierPartCharacters;
}

See also

RuleAttributes
Name
RuleExpression
SimpleRule
SimpleRuleBody
token vs. regex