title: Using Scanner/Parser/Lexer for script collation
body: <p>I'm working on a JavaScript collator/compositor implemented in Java. It works, but there has to be a better way to implement it and I think a Lexer may be the way forward, but I'm a little fuzzy.</p>
  
  <p>I've developed a meta syntax for the compositor which is a subset of the JavaScript language.  As far as a typical JavaScript interpreter is concerned, the compositor meta syntax is legal, just not functional (I'm using synonyms to reserved words as labels followed by code blocks which the compositor is supposed to interpret). Right now, I'm using a scanner and regex to find the meta syntax in source files, then do a shallow lexical transform based on detection of legal expressions.</p>
  
  <p>There is a tight coupling between the rewritten javascript and the scanner/parser which I am not happy with, as the rewritten javascript uses features of an object support library specially written for the purpose, and that library is subject to change.</p>
  
  <p>I'm hoping that I can declare just the meta syntax in Backaus-Naur or EBNF, feed it to a lexer (ANTRL?), and on the basis of meta syntax expressions detected in source files, direct the compositor to certain actions, such as prepending a required script to another, declaring a variable, generating text for a suitably parameterised library function invocation, or even compressing a script.</p>
  
  <p>Is this the appropriate way to make a compositor? Should I even be using a Scanner/Parser/Lexer approach to compositing JavaScript? Any feedback appreciated- I'm not quite sure where to start :)</p>
  
  <p>UPDATE:
  Here is a more concrete example- sample object declaration with meta syntax:</p>
  
  <pre><code>namespace: ie.ondevice
  {
      use: ie.ondevice.lang.Mixin;
      use: ie.ondevice.TraitsDeclaration;
  
      declare: Example &lt; Mixin | TraitsDeclaration
      {
          include: "path/to/file.extension";
          // implementation here
      }
   }
  </code></pre>
  
  <p>This describes the object ie.ondevice.Example, that inherits Mixin and resembles (i.e. 'implements the same functions and traits as') TraitsDeclaration. The compositor would detect use statements, and fail if the namespace does not map to a valid file location, or otherwise prepend the scripts in which object declarations reside, preprocessing meta syntax there before collation.</p>
  
  <p>The rewrite rules, expressed in terms of the object support library I mentioned would result in a file that <strong>may</strong> look like this (I've developed a number of ways to express the object):</p>
  
  <pre><code>module("ie.ondevice.Example", function (mScope)
  {
     // mScope is a delegate
     mScope.use("ie.ondevice.lang.Mixin");
     mScope.use("ie.ondevice.TraitsDeclaration");
  
     // As a result of two use statements, the mScope.localVars string would
     // would look like this: "var Mixin= ie.ondevice.lang.Mixin, TraitsDeclaration= ie.ondevice.TraitsDeclaration
     // by evaling we introduce 'imported' objects with their 'local'names
     eval(mScope.localVars); 
  
     // Function.prototype has been extended with the functions
     // inherits, define, defineStatic, resembles and getName
  
     // Prototypal inheritance using an anonymous bridge constructor
     Example.inherits(Mixin);
  
     // named methods and properties are added to Example.prototype
     Example.define
     (
         // functions and other properties
     );
     // ensures that Example.prototype has all the same
     // property names and types as TraitsDeclaration.prototype
     // throwing an exception if not the case.
     // This is optionally turned off for production- these
     // operations are only performed when the object is declared
     // - instantiation incurs no additional overhead
     Example.resembles(TraitsDeclaration);
  
     // constructor
     function Example ()
     {
         Mixin.call(this);
     };
  
     // will generate the ie.ondevice object hierarchy if required
     // and avail the constructor to it
     mScope.exports(Example);
   });
  </code></pre>
  
  <p>Perhaps I'm over architecting my requirements, but what I would really like is an event driven collator- listeners could then be loosely coupled to directive detections.</p>