Child pages
  • TokenRule
Skip to end of metadata
Go to start of metadata

TokenRule is the same as SimpleRule with the exception of the following:

  • The rules declared through TokenRule describe a language token.
  • TokenRule type rules are automatically marked with the ExplicitSpaces attribute which indicates that whitespace rules should be placed manually.

TokenRule is specifically treated during error recovery. Besides, the parser will point at the rule, not at its content.

A token may also be described by RegexRule, however the structure of such rules won’t be accessible, which can compromise code highlighting and error recovery. As an example, comments and strings should be described with TokenRule, otherwise the parser will not be able to spot an unclosed string or comment.

For example:

See also

token vs. regex

  • No labels