title: Using HTMLParser in Python efficiently
body: <p>In response to <a href="https://stackoverflow.com/questions/4182331">Python regular expression</a> I tried to implement an HTML parser using <code>HTMLParser</code>:</p>
  
  <pre><code>import HTMLParser
  
  class ExtractHeadings(HTMLParser.HTMLParser):
  
    def __init__(self):
      HTMLParser.HTMLParser.__init__(self)
      self.text = None
      self.headings = []
  
    def is_relevant(self, tagname):
      return tagname == 'h1' or tagname == 'h2'
  
    def handle_starttag(self, tag, attrs):
      if self.is_relevant(tag):
        self.in_heading = True
        self.text = ''
  
    def handle_endtag(self, tag):
      if self.is_relevant(tag):
        self.headings += [self.text]
        self.text = None
  
    def handle_data(self, data):
      if self.text != None:
        self.text += data
  
    def handle_charref(self, name):
      if self.text != None:
        if name[0] == 'x':
          self.text += chr(int(name[1:], 16))
        else:
          self.text += chr(int(name))
  
    def handle_entityref(self, name):
      if self.text != None:
        print 'TODO: entity %s' % name
  
  def extract_headings(text):
    parser = ExtractHeadings()
    parser.feed(text)
    return parser.headings
  
  print extract_headings('abdk3&lt;h1&gt;The content we need&lt;/h1&gt;aaaaabbb&lt;h2&gt;The content we need2&lt;/h2&gt;')
  print extract_headings('before&lt;h1&gt;&amp;#72;e&amp;#x6c;&amp;#108;o&lt;/h1&gt;after')
  </code></pre>
  
  <p>Doing that I wondered if the API of this module is bad or if I didn't notice some important things. My questions are:</p>
  
  <ul>
  <li>Why does my implementation of <code>handle_charref</code> have to be that complex? I would have expected that a good API passes the codepoint as a parameter, not either <code>x6c</code> or <code>72</code> as string.</li>
  <li>Why doesn't the default implementation of <code>handle_charref</code> call <code>handle_data</code> with an appropriate string?</li>
  <li>Why is there no utility implementation of <code>handle_entityref</code> that I could just call? It could be named <code>handle_entityref_HTML4</code> and would lookup the entities defined in HTML 4 and then call <code>handle_data</code> on them.</li>
  </ul>
  
  <p>If that API were provided, writing custom HTML parsers would be much easier. So where is my misunderstanding?</p>