title: python: replacing regex with BNF or pyparsing
body: <p>I am parsing a relatively simple text, where each line describes a game unit. I have little knowledge of parsing techniques, so I used the following ad hoc solution:</p>
  
  <pre><code>class Unit:
      # rules is an ordered dictionary of tagged regex that is intended to be applied in the given order
      # the group named V would correspond to the value (if any) for that particular tag
      rules = (
          ('Level', r'Lv. (?P&lt;V&gt;\d+)'),
          ('DPS', r'DPS: (?P&lt;V&gt;\d+)'),
          ('Type', r'(?P&lt;V&gt;Tank|Infantry|Artillery'),
          #the XXX will be expanded into a list of valid traits
          #note: (XXX| )* wouldn't work; it will match the first space it finds,
          #and stop at that if it's in front of something other than a trait
          ('Traits', r'(?P&lt;V&gt;(XXX)(XXX| )*)'),
          # flavor text, if any, ends with a dot
          ('FlavorText', r'(?P&lt;V&gt;.*\."?$)'),
          )
      rules = collections.OrderedDict(rules)
      traits = '|'.join('All-Terrain', 'Armored', 'Anti-Aircraft', 'Motorized')
      rules['Traits'] = re.sub('XXX', effects, rules['Traits'])
  
      for x in rules:
          rules[x] = re.sub('&lt;V&gt;', '&lt;'+x+'&gt;', rules[x])
          rules[x] = re.compile(rules[x])
  
      def __init__(self, data)
          # data looks like this:
          # Lv. 5 Tank DPS: 55 Motorized Armored
          for field, regex in Item.rules.items():
              data = regex.sub(self.parse, data, 1)
          if data:
              raise ParserError('Could not parse part of the input: ' + data)
  
      def parse(self, m):
          if len(m.groupdict()) != 1:
              Exception('Expected a single named group')
          field, value = m.groupdict().popitem()
          setattr(self, field, value)
          return ''
  </code></pre>
  
  <p>It works fine, but I feel I reached the limit of regex power. Specifically, in the case of Traits, the value ends up being a string that I need to split and convert into a list at a later point: e.g., obj.Traits would be set to 'Motorized Armored' in this code, but in a later function changed to ('Motorized', 'Armored').</p>
  
  <p>I'm thinking of converting this code to use either EBNF or pyparsing grammar or something like that. My goals are:</p>
  
  <ul>
  <li>make this code neater and less error-prone</li>
  <li>avoid the ugly treatment of the case with a list of values (where I need do replacement inside the regex first, and later post-process the result to convert a string into a list)</li>
  </ul>
  
  <p>What would be your suggestions about what to use, and how to rewrite the code?</p>
  
  <p>P.S. I skipped some parts of the code to avoid clutter; if I introduced any errors in the process, sorry - the original code does work :)</p>