title: Performance of tokenizing CSS in PHP
body: <p><em>This is a noob question from someone who hasn't written a parser/lexer ever before.</em></p>
  
  <p>I'm writing a tokenizer/parser for CSS in PHP (please don't repeat with 'OMG, why in PHP?'). The syntax is written down by the W3C neatly <a href="http://www.w3.org/TR/CSS2/grammar.html" rel="nofollow noreferrer">here (CSS2.1)</a> and <a href="http://www.w3.org/TR/2003/WD-css3-syntax-20030813/#tokenization" rel="nofollow noreferrer">here (CSS3, draft)</a>.</p>
  
  <p>It's a list of 21 possible tokens, that all (but two) cannot be represented as static strings.</p>
  
  <p>My current approach is to loop through an array containing the 21 patterns over and over again, do an <code>if (preg_match())</code> and reduce the source string match by match. In principle this works really good. However, for a 1000 lines CSS string this takes something between 2 and 8 seconds, which is too much for my project.</p>
  
  <p>Now I'm banging my head how other parsers tokenize <em>and</em> parse CSS in fractions of seconds. OK, C <em>is</em> always faster than PHP, but nonetheless, are there any obvious <em>D'Oh!</em> s that I fell into?</p>
  
  <p>I made some optimizations, like checking for '@', '#' or '"' as the first char of the remaining string and applying only the relevant regexp then, but this hadn't brought any great performance boosts.</p>
  
  <p>My code (snippet) so far:</p>
  
  <pre><code>$TOKENS = array(
    'IDENT' =&gt; '...regexp...',
    'ATKEYWORD' =&gt; '@...regexp...',
    'String' =&gt; '"...regexp..."|\'...regexp...\'',
    //...
  );
  
  $string = '...CSS source string...';
  $stream = array();
  
  // we reduce $string token by token
  while ($string != '') {
      $string = ltrim($string, " \t\r\n\f"); // unconsumed whitespace at the
          // start is insignificant but doing a trim reduces exec time by 25%
      $matches = array();
      // loop through all possible tokens
      foreach ($TOKENS as $t =&gt; $p) {
          // The '&amp;' is used as delimiter, because it isn't used anywhere in
          // the token regexps
          if (preg_match('&amp;^'.$p.'&amp;Su', $string, $matches)) {
              $stream[] = array($t, $matches[0]);
              $string = substr($string, strlen($matches[0]));
              // Yay! We found one that matches!
              continue 2;
          }
      }
      // if we come here, we have a syntax error and handle it somehow
  }
  
  // result: an array $stream consisting of arrays with
  // 0 =&gt; type of token
  // 1 =&gt; token content
  </code></pre>