title: Lexical Analyser In Java
body: <p>I have been trying to write a simple lexical analyzer in java . </p>
  
  <p>The File <strong>Token.java</strong> looks as follows : </p>
  
  <pre><code>import java.util.regex.Matcher;
  import java.util.regex.Pattern;
  
  public enum Token {
  
      TK_MINUS ("-"), 
      TK_PLUS ("\\+"), 
      TK_MUL ("\\*"), 
      TK_DIV ("/"), 
      TK_NOT ("~"), 
      TK_AND ("&amp;"),  
      TK_OR ("\\|"),  
      TK_LESS ("&lt;"),
      TK_LEG ("&lt;="),
      TK_GT ("&gt;"),
      TK_GEQ ("&gt;="), 
      TK_EQ ("=="),
      TK_ASSIGN ("="),
      TK_OPEN ("\\("),
      TK_CLOSE ("\\)"), 
      TK_SEMI (";"), 
      TK_COMMA (","), 
      TK_KEY_DEFINE ("define"), 
      TK_KEY_AS ("as"),
      TK_KEY_IS ("is"),
      TK_KEY_IF ("if"), 
      TK_KEY_THEN ("then"), 
      TK_KEY_ELSE ("else"), 
      TK_KEY_ENDIF ("endif"),
      OPEN_BRACKET ("\\{"),
      CLOSE_BRACKET ("\\}"),
      DIFFERENT ("&lt;&gt;"),
  
      STRING ("\"[^\"]+\""),
      INTEGER ("\\d"), 
      IDENTIFIER ("\\w+");
  
      private final Pattern pattern;
  
      Token(String regex) {
          pattern = Pattern.compile("^" + regex);
      }
  
      int endOfMatch(String s) {
          Matcher m = pattern.matcher(s);
  
          if (m.find()) {
              return m.end();
          }
          return -1;
      }
  }
  </code></pre>
  
  <p>The Lexer is as follows : <strong>Lexer.java</strong></p>
  
  <pre><code>import java.io.IOException;
  import java.nio.file.Files;
  import java.nio.file.Paths;
  import java.util.HashSet;
  import java.util.Set;
  import java.util.stream.Stream;
  
  public class Lexer {
      private StringBuilder input = new StringBuilder();
      private Token token;
      private String lexema;
      private boolean exausthed = false;
      private String errorMessage = "";
      private Set&lt;Character&gt; blankChars = new HashSet&lt;Character&gt;();
  
      public Lexer(String filePath) {
          try (Stream&lt;String&gt; st = Files.lines(Paths.get(filePath))) {
              st.forEach(input::append);
          } catch (IOException ex) {
              exausthed = true;
              errorMessage = "Could not read file: " + filePath;
              return;
          }
  
          blankChars.add('\r');
          blankChars.add('\n');
          blankChars.add((char) 8);
          blankChars.add((char) 9);
          blankChars.add((char) 11);
          blankChars.add((char) 12);
          blankChars.add((char) 32);
  
          moveAhead();
      }
  
      public void moveAhead() {
          if (exausthed) {
              return;
          }
  
          if (input.length() == 0) {
              exausthed = true;
              return;
          }
  
          ignoreWhiteSpaces();
  
          if (findNextToken()) {
              return;
          }
  
          exausthed = true;
  
          if (input.length() &gt; 0) {
              errorMessage = "Unexpected symbol: '" + input.charAt(0) + "'";
          }
      }
  
      private void ignoreWhiteSpaces() {
          int charsToDelete = 0;
  
          while (blankChars.contains(input.charAt(charsToDelete))) {
              charsToDelete++;
          }
  
          if (charsToDelete &gt; 0) {
              input.delete(0, charsToDelete);
          }
      }
  
      private boolean findNextToken() {
          for (Token t : Token.values()) {
              int end = t.endOfMatch(input.toString());
  
              if (end != -1) {
                  token = t;
                  lexema = input.substring(0, end);
                  input.delete(0, end);
                  return true;
              }
          }
  
          return false;
      }
  
      public Token currentToken() {
          return token;
      }
  
      public String currentLexema() {
          return lexema;
      }
  
      public boolean isSuccessful() {
          return errorMessage.isEmpty();
      }
  
      public String errorMessage() {
          return errorMessage;
      }
  
      public boolean isExausthed() {
          return exausthed;
      }
  }
  </code></pre>
  
  <p>And can be tested with a <strong>Try.java</strong> as follows : </p>
  
  <pre><code>public class Try {
  
      public static void main(String[] args) {
  
          Lexer lexer = new Lexer("C:/Users/Input.txt");
  
          System.out.println("Lexical Analysis");
          System.out.println("-----------------");
          while (!lexer.isExausthed()) {
              System.out.printf("%-18s :  %s \n",lexer.currentLexema() , lexer.currentToken());
              lexer.moveAhead();
          }
  
          if (lexer.isSuccessful()) {
              System.out.println("Ok! :D");
          } else {
              System.out.println(lexer.errorMessage());
          }
      }
  }
  </code></pre>
  
  <p>Say the <strong>Input.txt</strong> has </p>
  
  <pre><code>define mine 
  a=1000;
  b=23.5;
  </code></pre>
  
  <p>The <strong>output I expect is</strong> </p>
  
  <pre><code>define : TK_KEYWORD
  mine : IDENTIFIER
  a : IDENTIFIER
  = : TK_ASSIGN
  1000 : INTEGER
  ; : TK_SEMI
  b : IDENTIFIER
  = : TK_ASSIGN
  23.5 : REAL
  </code></pre>
  
  <p>But The issue I am facing is : It treats each digit like </p>
  
  <pre><code>1 INTEGER
  0 INTEGER
  0 INTEGER
  0 INTEGER
  </code></pre>
  
  <p>also it doesn't recognize Real numbers . I get:</p>
  
  <blockquote>
    <p>Unexpected symbol: '.'</p>
  </blockquote>
  
  <p>What are the changes needed to get the expected results?</p>