title: ANTLR Token recognition error with lexer and parser
body: <p>I am writing an ANTLR Lexer and Parser grammar that will parse text that is quite similar to a Java class. Eventually it will parse text like the following:</p>
  
  <pre><code>reference schema:"https://schema.org/";
  reference dc:"https://www.dublincore.org/";
  
  type dc:Author {
  }
  </code></pre>
  
  <p>I am building up the Lexer and Parser slowly. I have successfully managed to parse the <code>reference</code>s but have hit a wall when parsing the <code>type</code>. </p>
  
  <p>Before adding support for the <code>type</code> I was able to use string literals for space, colon, and semi-colon in the parser but after I encountered <code>cannot create implicit token for string literal</code> errors. I defined a lexer rule for each of those characters and replaced all occurrences of the literal with the rule. However this broke the parsing of <code>reference</code>s.</p>
  
  <p>I have included my lexer and parser that successfully parses <code>reference</code>s below (along with a sample input and the parsed abstract syntax tree) and the evolved versions which isn't working. I am not getting any compilation errors but plenty of <code>token recognition error</code>s (screenshot included below).</p>
  
  <p>What is the correct way to handle the parsing?</p>
  
  <h1>Working</h1>
  
  <h2>Lexer</h2>
  
  <pre><code>lexer grammar WorkingLexerGrammar;
  
  WS: ('\t' | '\n' | '\r' )+ -&gt; skip ;
  
  fragment Colon : ':';
  fragment SemiColon: ';';
  fragment Underscores: '_'+ ;
  fragment Digits: [0-9]+ ;
  fragment LowercaseLetters: [a-z]+ ;
  fragment UppercaseLetters: [A-Z]+ ;
  fragment String: '"' .*? '"' ;
  fragment Prefix: (Underscores | Digits | LowercaseLetters)+ ;
  
  REFERENCE_KEYWORD: 'reference' ;
  TYPE_KEYWORD: 'type' ;
  
  PREFIXED_REFERENCE: ' ' -&gt; pushMode(PrefixedReferenceMode) ;
  
  mode PrefixedReferenceMode;
  REFERENCE_PREFIX: Prefix;
  REFERENCE_PREFIX_SEPARATOR: ':' -&gt; pushMode(IriMode);
  END_IRI: ';' -&gt; popMode;
  
  mode IriMode;
  IRI: String  -&gt; popMode;
  </code></pre>
  
  <h2>Parser</h2>
  
  <pre><code>parser grammar WorkingParserGrammar ;
  
  options { tokenVocab=WorkingLexerGrammar; }
  
  document: reference* EOF ;
  
  prefixedReference: REFERENCE_PREFIX ':' IRI;
  reference: REFERENCE_KEYWORD ' ' prefixedReference ';';
  </code></pre>
  
  <h2>Input</h2>
  
  <pre><code>reference schema:"https://schema.org/";
  reference dc:"https://www.dublincore.org/";
  </code></pre>
  
  <h2>Output</h2>
  
  <p><a href="https://i.stack.imgur.com/6yxeJ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/6yxeJ.png" alt="Abstract syntax tree output"></a></p>
  
  <h1>Evolved (not working)</h1>
  
  <h2>Lexer</h2>
  
  <pre><code>lexer grammar NotWorkingLexerGrammar;
  
  WS: ('\t' | '\n' | '\r' )+ -&gt; skip ;
  
  fragment Colon : ':';
  fragment SemiColon: ';';
  fragment Underscores: '_'+ ;
  fragment Digits: [0-9]+ ;
  fragment LowercaseLetters: [a-z]+ ;
  fragment UppercaseLetters: [A-Z]+ ;
  fragment String: '"' .*? '"' ;
  fragment Prefix: (Underscores | Digits | LowercaseLetters)+ ;
  
  COLON: Colon;
  SEMICOLON: SemiColon;
  SPACE: ' ';
  
  REFERENCE_KEYWORD: 'reference' ;
  TYPE_KEYWORD: 'type' ;
  
  PREFIXED_REFERENCE: SPACE -&gt; pushMode(PrefixedReferenceMode) ;
  
  mode PrefixedReferenceMode;
  REFERENCE_PREFIX: Prefix;
  REFERENCE_PREFIX_SEPARATOR: COLON -&gt; pushMode(IriMode);
  END_IRI: SEMICOLON -&gt; popMode;
  
  mode IriMode;
  IRI: String  -&gt; popMode;
  
  PREFIXED_NAME: SPACE -&gt; pushMode(PrefixedNameMode) ;
  
  mode PrefixedNameMode;
  NAME_PREFIX: Prefix;
  NAME_PREFIX_SEPARATOR: COLON -&gt; pushMode(LocalNameMode);
  END_NAME: SEMICOLON -&gt; popMode;
  
  mode LocalNameMode;
  LOCAL_NAME: (Underscores | Digits | LowercaseLetters | UppercaseLetters)+  -&gt; popMode;
  </code></pre>
  
  <h2>Parser</h2>
  
  <pre><code>parser grammar NotWorkingParserGrammar ;
  
  options { tokenVocab=NotWorkingLexerGrammar; }
  
  document: reference* type* EOF ;
  
  prefixedReference: REFERENCE_PREFIX COLON IRI;
  reference: REFERENCE_KEYWORD SPACE prefixedReference SEMICOLON;
  
  prefixedName: NAME_PREFIX SPACE LOCAL_NAME;
  type: TYPE_KEYWORD SPACE prefixedName;
  </code></pre>
  
  <h2>Output</h2>
  
  <p><a href="https://i.stack.imgur.com/iynRI.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/iynRI.png" alt="Abstract syntax tree not parsing correctly"></a></p>
  
  <hr>
  
  <p>Following <a href="https://stackoverflow.com/a/61665080/336752">Bart Kiers' help</a> I have made two updates to the lexer and parser grammars with varying success.</p>
  
  <h1>First update</h1>
  
  <p>This change parses the type definition correctly but only if I remove the lexer rules for reference. I think the reason for that is that the two rules are the same (i.e. <code>PREFIXED_REFERENCE: SPACE -&gt; pushMode(PrefixedReferenceMode) ;</code> for reference and <code>PREFIXED_NAME: SPACE -&gt; pushMode(PrefixedNameMode) ;</code> for type) – that is they both match on a space. My second update attempts to fix this but the full lexer and parser grammars are below.</p>
  
  <h2>Lexer</h2>
  
  <pre><code>lexer grammar NotWorkingLexerGrammar;
  
  WS: ('\t' | '\n' | '\r' )+ -&gt; skip ;
  
  fragment Underscores: '_'+ ;
  fragment Digits: [0-9]+ ;
  fragment LowercaseLetters: [a-z]+ ;
  fragment UppercaseLetters: [A-Z]+ ;
  fragment String: '"' .*? '"' ;
  fragment Prefix: (Underscores | Digits | LowercaseLetters)+ ;
  
  fragment COLON: ':';
  fragment SEMICOLON: ';';
  fragment SPACE: ' ';
  
  fragment REFERENCE_KEYWORD: 'reference' ;
  fragment TYPE_KEYWORD: 'type' ;
  
  PREFIXED_REFERENCE: SPACE -&gt; pushMode(PrefixedReferenceMode) ;
  
  mode PrefixedReferenceMode;
  REFERENCE_PREFIX: Prefix;
  REFERENCE_PREFIX_SEPARATOR: COLON -&gt; pushMode(IriMode);
  END_IRI: SEMICOLON -&gt; popMode;
  
  mode IriMode;
  IRI: String  -&gt; popMode;
  
  PREFIXED_NAME: SPACE -&gt; pushMode(PrefixedNameMode) ;
  
  mode PrefixedNameMode;
  NAME_PREFIX: Prefix;
  NAME_PREFIX_SEPARATOR: COLON -&gt; pushMode(LocalNameMode);
  END_NAME: SEMICOLON -&gt; popMode;
  
  mode LocalNameMode;
  LOCAL_NAME: (Underscores | Digits | LowercaseLetters | UppercaseLetters)+  -&gt; popMode;
  </code></pre>
  
  <h2>Parser</h2>
  
  <pre><code>parser grammar NotWorkingParserGrammar ;
  
  options { tokenVocab=NotWorkingLexerGrammar; }
  
  document: reference* type* EOF ;
  
  prefixedReference: REFERENCE_PREFIX REFERENCE_PREFIX_SEPARATOR IRI;
  reference: REFERENCE_KEYWORD PREFIXED_REFERENCE prefixedReference END_IRI;
  
  prefixedName: NAME_PREFIX NAME_PREFIX_SEPARATOR LOCAL_NAME;
  type: TYPE_KEYWORD PREFIXED_NAME prefixedName END_NAME;
  </code></pre>
  
  <h1>Second update</h1>
  
  <p>In an attempt to fix this I moved the <code>reference</code> and <code>type</code> keywords to the Lexer rules for the corresponding parts but this only parses the type if I remove all of the Lexer rules for reference. However references are parsed correctly.</p>
  
  <h2>Lexer</h2>
  
  <pre><code>lexer grammar NotWorkingLexerGrammar;
  
  WS: ('\t' | '\n' | '\r' )+ -&gt; skip ;
  
  fragment Underscores: '_'+ ;
  fragment Digits: [0-9]+ ;
  fragment LowercaseLetters: [a-z]+ ;
  fragment UppercaseLetters: [A-Z]+ ;
  fragment String: '"' .*? '"' ;
  fragment Prefix: (Underscores | Digits | LowercaseLetters)+ ;
  
  fragment COLON: ':';
  fragment SEMICOLON: ';';
  fragment SPACE: ' ';
  
  fragment REFERENCE_KEYWORD: 'reference' ;
  fragment TYPE_KEYWORD: 'type' ;
  
  PREFIXED_REFERENCE: REFERENCE_KEYWORD SPACE -&gt; pushMode(PrefixedReferenceMode) ;
  
  mode PrefixedReferenceMode;
  REFERENCE_PREFIX: Prefix;
  REFERENCE_PREFIX_SEPARATOR: COLON -&gt; pushMode(IriMode);
  END_IRI: SEMICOLON -&gt; popMode;
  
  mode IriMode;
  IRI: String  -&gt; popMode;
  
  TYPE_DEFINITION: TYPE_KEYWORD SPACE -&gt; pushMode(PrefixedNameMode) ;
  
  mode PrefixedNameMode;
  NAME_PREFIX: Prefix;
  NAME_PREFIX_SEPARATOR: COLON -&gt; pushMode(LocalNameMode);
  END_NAME: SEMICOLON -&gt; popMode;
  
  mode LocalNameMode;
  LOCAL_NAME: (Underscores | Digits | LowercaseLetters | UppercaseLetters)+  -&gt; popMode;
  </code></pre>
  
  <h2>Parser</h2>
  
  <pre><code>parser grammar NotWorkingParserGrammar ;
  
  options { tokenVocab=NotWorkingLexerGrammar; }
  
  document: reference* type* EOF ;
  
  prefixedReference: REFERENCE_PREFIX REFERENCE_PREFIX_SEPARATOR IRI;
  reference: PREFIXED_REFERENCE prefixedReference END_IRI;
  
  prefixedName: NAME_PREFIX NAME_PREFIX_SEPARATOR LOCAL_NAME;
  type: TYPE_DEFINITION prefixedName END_NAME;
  </code></pre>
  
  <h2>Output</h2>
  
  <p>For the following input:</p>
  
  <pre><code>reference schema:"https://schema.org/";
  reference dc:"https://www.dublincore.org/";
  
  type dc:Author;
  </code></pre>
  
  <p>This is the output:</p>
  
  <pre><code>line 4:0 token recognition error at: 't'
  line 4:1 token recognition error at: 'y'
  line 4:2 token recognition error at: 'p'
  line 4:3 token recognition error at: 'e'
  line 4:4 token recognition error at: ' '
  line 4:5 token recognition error at: 'd'
  line 4:6 token recognition error at: 'c'
  line 4:7 token recognition error at: ':'
  line 4:8 token recognition error at: 'A'
  line 4:9 token recognition error at: 'u'
  line 4:10 token recognition error at: 't'
  line 4:11 token recognition error at: 'h'
  line 4:12 token recognition error at: 'o'
  line 4:13 token recognition error at: 'r;'
  </code></pre>
  
  <p>My reasoning for using modes is to limit the scope of rules. This is a language I control but would prefer not to change it dramatically. There is much more to the language than I've shown here and we have already have a grammar (currently a combined grammar) but it is quite brittle. I tried to make a change to prevent uppercase characters in prefixes but permit them in the local name but this snowballed and other rules started applying. Research suggested that modes was an approach to handle this situation but I'm not very familiar with ANTLR so I've possibly misunderstood it.</p>
  
  <p><a href="https://i.stack.imgur.com/1lRIb.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/1lRIb.png" alt="Abstract syntax tree for second update"></a></p>