TokenRule is the same as
SimpleRule with the exception of the following:
- The rules declared through
TokenRuledescribe a language token.
TokenRuletype rules are automatically marked with the
ExplicitSpacesattribute which indicates that whitespace rules should be placed manually.
TokenRule is specifically treated during error recovery. Besides, the parser will point at the rule, not at its content.
A token may also be described by
RegexRule, however the structure of such rules won’t be accessible, which can compromise code highlighting and error recovery. As an example, comments and strings should be described with
TokenRule, otherwise the parser will not be able to spot an unclosed string or comment.