TokenRule
is the same as SimpleRule
with the exception of the following:
- The rules declared through
TokenRule
describe a language token. TokenRule
type rules are automatically marked with theExplicitSpaces
attribute which indicates that whitespace rules should be placed manually.
TokenRule
is specifically treated during error recovery. Besides, the parser will point at the rule, not at its content.
A token may also be described by RegexRule
, however the structure of such rules won’t be accessible, which can compromise code highlighting and error recovery. As an example, comments and strings should be described with TokenRule
, otherwise the parser will not be able to spot an unclosed string or comment.
For example:
See also
RuleAttributes
Name
RuleExpression
SimpleRule
SimpleRuleBody
token vs. regex